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Overview 


Many incoming college students are referred to remedial programs in math or English 
based on scores they earn on standardized placement tests. Yet research shows that some 
students assigned to remediation based on test scores would likely succeed in a college-level 
course in the same subject area without first taking a remedial course if given that opportunity. 
Research also suggests that other measures of student skills and performance, and in particular 
high school grade point average (GPA), may be useful in assessing college readiness. 


CAPR is conducting a random assignment study of a multiple measures placement 
system based on data analytics to determine whether it yields placement determinations that 
lead to better student outcomes than a system based on test scores alone. Seven community 
colleges in the State University of New York (SUNY) system are participating in the study. 
The alternative placement system we evaluate uses data on prior students to weight multiple 
measures — including both placement test scores and high school GPAs — in predictive 
algorithms developed at each college that are then used to place incoming students into 
remedial or college-level courses. Over 13,000 incoming students who arrived at these 
colleges in the fall 2016, spring 2017, and fall 2017 terms were randomly assigned to be 
placed using either the status quo placement system (the control group) or the alternative 
placement system (the program group). The three cohorts of students will be tracked through 
the fall 2018 term, resulting in the collection of three to five semesters of outcomes data, 
depending on the cohort. 


This interim report, the first of two, examines implementation of the alternative 
placement system at the colleges and presents results on first-term impacts for 4,729 students 
in the fall 2016 cohort. The initial results are promising. The early findings show that: 


e While implementing the alternative system was more complex than 
expected, every college developed the procedures that were required to 
make it work as intended. 


e Many program group students were placed differently than they would have 
been under the status quo placement system. In math, 14 percent of program 
group students placed higher than they would have under a test-only system 
(i.e., in college-level), while 7 percent placed lower (i.e., in remedial). In 
English, 41.5 percent placed higher, while 6.5 percent placed lower. 


e Program group students were 3.1 and 12.5 percentage points more likely 
than control group students to both enroll in and complete (with a grade of 
C or higher) a college-level math or English course in the first term. 


ill 


(Enrollment and completion rates among the control group were 14.1 
percent in math and 27.2 percent in English.) 


e Women appeared to benefit more than men from program group status in 
math on college-level math course placement, enrollment, and completion 
(with a grade of C or higher) outcomes; Black and Hispanic students 
appeared to benefit more than White students from program group status 
in English on college-level English course placement and enrollment 
outcomes, but not on completion (with a grade of C or higher). 


e Implementation of the alternative system added roughly $110 per student 
to status quo fall-term costs for testing and placing students at the colleges; 
ongoing costs in the subsequent fall term were roughly $40 per student 
above status quo costs. 


The final report, to be released in 2019, will examine a range of student outcomes for 
all three cohorts, including completion of introductory college-level courses, persistence, and 
the accumulation of college credits over the long term. 
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Executive Summary 


Two thirds of students who attend community colleges and two fifths of students who 
attend public four-year colleges enroll in one or more remedial courses (also known as 
developmental education courses) to strengthen their skills for college-level coursework 
(Chen, 2016). Remedial courses may be helpful to some students, but they also require 
students to make a substantial investment of limited time and money that could otherwise be 
applied to college-level coursework, and studies suggest that the effects of remedial courses 
on student outcomes are at best mixed for those who are thought to be on the cusp of needing 
additional academic support (Jaggars & Stacey, 2014). Further, students who start college in 
remedial coursework are less likely to graduate (Attewell, Lavin, Domina, & Levey, 2006). It 
is therefore important to decide which incoming students ought to enroll in remedial courses. 


Currently, most students who participate in remediation in math or English (or both) 
are referred to these programs based on the scores they earn on standardized placement tests, 
which they typically take when they arrive at college. Yet in recent years, questions have 
arisen about how useful these standardized tests are for placing incoming students into 
remedial and college-level coursework. Research shows that some students assigned to 
remediation based on test scores would likely pass a college-level course in the same subject 
area without first taking a remedial course if given that opportunity; it also suggests that using 
multiple measures of student skills and performance, and in particular high school grade point 
average (GPA), may be useful in assessing college readiness (Belfield & Crosta, 2012; Scott- 
Clayton, 2012). 


Partly in response to these findings, an increasing number of colleges are now 
exploring or beginning to use multiple measures to place incoming students into remedial or 
college-level courses (Rutschow & Mayer, 2018). Multiple measures placement systems 
often make use of placement test results but also consider other relevant data on incoming 
students, such as high school GPA. While studies suggest that using multiple measures may 
result in the improved placement of students into remedial and college-level courses, little 
evidence to date has shown that using a multiple measures placement system influences other 
student outcomes. 


To address this gap, CAPR is conducting a random assignment study of a multiple 
measures placement system to determine whether it yields placement determinations that lead 
to better student outcomes than a system based on test scores alone. Seven community 
colleges in the State University of New York (SUNY) system are participating in the study. 
The placement system CAPR researchers are evaluating uses data on prior students to develop 
predictive algorithms at each college to weight multiple measures — including placement 


test scores, high school GPA, years since high school graduation, and in some cases other 
measures — that are then used to place incoming students into remedial or college-level 
courses. Over 13,000 incoming students who arrived at these colleges in the fall 2016, spring 
2017, and fall 2017 terms were randomly assigned to be placed using either the status quo 
placement system (the control group) or the alternative placement system (the program 
group). The three cohorts of students will be tracked through the fall 2018 term, resulting in 
the collection of three to five semesters of outcomes data depending on the cohort. 


CAPR researchers and personnel from the seven colleges worked together to develop 
the data analytics algorithms and the alternative system for placement. Given differences 
among the SUNY community colleges participating in the study, the data analytics algorithms 
employed to assess program group students were created for each college individually (one 
each for math and English), using historical data from 201 1—14. Data on multiple measures — 
such as high school GPA, years since high school graduation, and placement test scores — as 
well as data on outcomes in college-level courses were used to create algorithms that weight 
each measure in the most appropriate way for predicting student performance in initial college- 
level math and English courses. 


After the algorithms were developed, historical data were also used to predict 
placement and success rates in initial college-level courses in each subject area at a range of 
cut points. Faculty at each college then created placement rules by choosing the cut points 
that would be used to place program group students into remedial or college-level math and 
English courses. 


Development of the algorithms using historical data showed that placement accuracy 
is a concern for all colleges in the study. Between one third and one half of prior students 
were estimated to have been “misplaced” in math and English at the colleges. Misplaced 
students include “underplaced” students, who were placed in a remedial course but would 
likely have been able to complete an initial college-level course with a grade of C or higher, 
as well as “overplaced” students, who were placed into and failed a college-level course. With 
one exception (math misplacement rates at one college), historical rates of underplacement 
were higher than historical rates of overplacement for both math and English at each of the 
colleges, and in most cases much higher. 


Implementation Findings 


The seven colleges in this study all followed very similar status quo placement 
procedures before beginning their involvement with this project. Most of the colleges relied 
heavily on the results of ACCUPLACER or other single tests for placement. CAPR research 
teams visited each of the seven participating colleges on two separate occasions to learn what 
college personnel thought about both the status quo and alternative placement systems and to 
better understand the processes required to implement the alternative system. 


While most interviewees at the colleges were quick to point out weaknesses in the 
status quo system, they also emphasized two strengths of that system: (1) the straightforward 
nature of comparing a student’s score on a test with an established cut score to place students 
(compared with the relative opacity of using the algorithm score produced under the 
alternative system, which combines weighted values from a number of different sources), and 
(2) the related efficiency of the status quo system, which allows students to be placed into 
coursework very quickly, and without need to obtain additional information. 


In terms of weaknesses, interviewees frequently reported their belief that the 
placement tests used under the status quo system were not doing a good job of placing 
students into the appropriate level of coursework. They also expressed strong concerns that 
students do not recognize how important the tests are and that some students proceed through 
the tests too quickly. 


Overall, implementation of the multiple measures, data analytics placement system 
created a significant amount of up-front work to develop new processes and procedures that, 
once in place, generally ran smoothly and with few problems. At the beginning of the project, 
colleges underwent a planning process of a year or more, in close collaboration with the 
research team, in order to make all of the changes required to implement the alternative 
placement system. 


Among other activities, each college did the following: (1) organized a group of 
people to take responsibility for developing the new system, (2) compiled a historical dataset 
in order to create the college’s algorithms, (3) developed or improved processes for obtaining 
high school transcripts for incoming students and for entering transcript information into IT 
systems in a useful way (which in some cases was time-consuming and challenging), (4) 
created procedures for uploading high school data into a data system where it could be 
combined with test data at the appropriate time, (5) changed IT systems to capture the 
placement determinations derived from the use of multiple measures, (6) created new 
placement reports for use by students and advisors, (7) provided training to testing staff and 
advisors on how to interpret the new placement determinations and communicate with 


students about them, and (8) conducted trial runs of the new processes to troubleshoot and 
avoid problems during actual implementation. 


While these activities were demanding, every college was successful in overcoming 
barriers and developing the procedures needed to support the operation of the data analytics 
placement system for its students. Five colleges achieved this benchmark in time for 
placement of students entering in the fall of 2016, while the other two colleges did so in time 
for new student intake in the fall of 2017. 


While many interviewees believed that the alternative system would place students 
more fairly and accurately, they also reported challenges and concerns. These issues largely 
involved: (1) undertaking such an extensive reform so quickly and establishing the buy-in to 
do so, (2) obtaining and entering large amounts of high school transcript data into the 
college’s computer system, (3) adjusting classroom and faculty assignments based on 
changed proportions of students in developmental and college-level courses, (4) not having 
placement information immediately available to students under the alternative system (in 
some cases, students had to wait a day or more to get their placement determinations), and 
(5) the potential limiting of access to support programs intended for underprepared (low- 
placing) students. 


Cost Findings 


We calculated costs for the five colleges participating in study intake for the fall 2016 
cohort using the ingredients method (Levin, McEwan, Belfield, Bowden, & Shand, 2017). 
Costs are derived from the inputs used at each college, multiplied by standardized prices per 
input. Relative to the status quo system, new resources were required to create the algorithms, 
to set up and administer the collection of data used in the algorithms, and to run the alternative 
system at the time of placement testing. Across the five colleges, implementation of the 
alternative placement system added $603,550 — or $110 per student — to status quo fall- 
term costs for testing and placing students. The per-student net implementation costs ranged 
from $70 to $320 at the different colleges, with lower costs generally associated with higher 
numbers of students at each college. More enrollments lead to lower costs per student because 
the costs of creating the algorithms for the new system are mostly fixed; they do not vary 
with the number of students involved. 


Ongoing costs in the subsequent fall term were much lower than the first-term 
implementation costs. Ongoing per-term costs were estimated at $215,300 — or $40 per 
student — above status quo costs. The per-student net ongoing costs ranged from $10 to $170 
at the different colleges. 


When information on the outcomes of the alternative placement system is available, 
cost estimates can be used as part of a cost-effectiveness analysis. Findings from such an 
analysis will be included in the final report. 


Placement Determinations of Program Group Students 


Because the multiple measures, data analytics placement system uses a different set of 
criteria than the status quo system, we might expect at least some changes in placement levels 
in math and English courses among program group students relative to what they would have 
been under the status quo. Importantly, however, any new placement procedure will not 
change the placement determinations of some students. Of the 2,455 students assigned to the 
program group, 92 percent took a placement test in math, and 76 percent took a placement test 
in English. Figure ES.1 shows how the placement determinations of such program students 
differed from what they would have been under the status quo. As expected, based on prior 
research, the proportion of higher placements outweighed the proportion of lower placements 
in both subject areas, particularly in English, where nearly half of program group students 
were placed differently than they would have been otherwise. 


Figure ES.1 


Observed Difference in Placement Relative to Status Quo Among Program Group 
Students Who Took a Placement Test in Each Subject Area 
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Early Impacts Findings 


In this experimental study, incoming students who took a placement test were 
randomly assigned to be placed using either the multiple measures, data analytics system or 
the status quo system. This assignment method creates two groups of students — program 
group and control group students — who should, in expectation, be similar in all ways other 
than their form of placement. The overall sample for our analysis of first-term outcomes 
consists of 4,729 students who took a placement test at the five colleges at the time of fall 
2016 entry, of whom 3,865, or about 82 percent, enrolled in at least one developmental or 
college-level course of any kind during the fall 2016 term. Because some students in the 
sample took either a math or an English placement test rather than both, the sample for our 
analysis of math outcomes is reduced to 4,371 students, and the sample for analysis of English 
outcomes is reduced to 3,533 students. We find that differences in student characteristics and 
in placement test scores between program and control group students are generally small and 
statistically insignificant, which provides reassurance that the randomized treatment 
procedures undertaken at the colleges were performed as intended. 


Our analyses were conducted using ordinary least squares regression models in which 
we controlled for college fixed effects and student characteristics such as gender, 
race/ethnicity, age, and financial aid status as well as proxies for college preparedness. 


For both math and English, we consider three outcomes as shown in Figure ES.2: the 
rate of college-level course placement (vs. remedial course placement) in the same subject 
area, the rate of college-level course enrollment in the same subject area, and the rate of 
college-level course completion with a grade of C or higher in the same subject area. 


As is shown, assignment to the program group produced positive and statistically 
significant effects on all three outcomes in both math and English. The impacts in English 
were substantially larger than the impacts in math. In math, students in the program group 
were, on average, 3.1 percentage points more likely to enroll in and complete (with a grade 
of C or higher) a college-level math course during their first term, after controlling for the 
full set of covariates. In English, students in the program group were 12.5 percentage points 
more likely to enroll in and complete a college-level English course. 


We also carried out analysis on the full sample to measure the effect that assignment 
to the program group had on earning college-level credits in any course or courses in the first 
term. Students in the program group earned, on average, 0.60 more college-level credits than 
students in the control group (p < .01; control group student students earned 5.17 credits, 
while program group students earned 5.77 credits). 


Figure ES.2 


College-Level Course Outcomes in Math and English 
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Finally, to examine whether program assignment led to differential first-term impacts 
by race/ethnicity (Black, Hispanic, White), Pell recipient status (yes, no), or gender (female, 
male), we conducted subgroup analyses and tested the significance of interaction effects for 
each subgroup. We limited these analyses to only those students who enrolled in any course 
at the college (because demographic information on students who did not enroll was 
unavailable), so the results of this analysis are not strictly causal. It is also worth noting that 
small sample sizes used in this first-term impacts analysis may limit the extent to which some 
subgroup effects are found to be statistically significant. 


In math, we find that most subgroups benefitted from program group status in terms 
of college-level math placement, enrollment, and enrollment and completion (with a grade of 
C or higher) outcomes (p < .1); the exceptions are that we find no statistically significant 
treatment impacts for men across all math outcomes considered and also find no statistically 
significant impacts on math course completion for Black and White students. 


Again in math, we find that interactions between the treatment status and each of the 
race/ethnicity and Pell recipient subgroups we considered are not statistically significant. This 
suggests that gaps in placement, enrollment, and completion rates in math between subgroups 
(other than the gender subgroups) may not have been affected by the treatment. We do find, 


however, that while men had higher math outcomes than women in both the control and 
program groups, women benefitted more from program group status in math on all three 
outcomes considered. For example, the male-female gap in the rate of enrollment in and 
completion (with a grade of C or higher) of college-level math narrowed from 4.5 percentage 
points among control group students to 0.4 percentage points among program group students. 
(The male control group rate was 19.5 percent.) 


In English, we find that all subgroups benefitted from program group status on all 
three outcomes considered (p < .01). Although significance testing on interaction effects in 
most cases failed to reveal differential impacts by subgroup, we do find evidence of 
differential treatment effects by racial/ethnic subgroup on two of the three considered 
outcomes. White students in the control group had higher English outcomes than Black and 
Hispanic students in the control group, but under program group status, the racial/ethnic gaps 
in both the rate of placement and the rate of enrollment in college-level English narrowed or 
even reversed. Yet we do not find evidence that program group status narrowed the gap in 
the rate of completion (with a grade of C or higher) of college-level English between White 
and Black or between White and Hispanic students. 


Looking Ahead 


These early results are broadly promising, but they are based on analyses of merely 
one semester of data. Additional impact analyses using data that are not yet available will be 
performed to further evaluate the effects of using a multiple measures, data analytics system 
to place incoming students. The final report from this study, to be released next year, will 
examine a range of student outcomes for all three cohorts for a period of three to five 
semesters after students’ initial entry into college at seven SUNY community colleges. 


Chapter 1 
Introduction 


Placement testing has become a near-universal part of the enrollment experience for 
incoming community college students (Bailey, Jaggars, & Jenkins, 2015). For decades, 
higher education institutions of all kinds have assessed the college readiness of incoming 
students. Selective institutions use admissions requirements to screen students, accepting or 
rejecting them on the basis of test scores, high school transcripts, and other application 
information (Cohen, Brawer, & Kisker, 2014). Open-access institutions — which include 
community colleges and some four-year institutions — accept all or most students for 
admission but then make a determination about whether or not those students are immediately 
ready for college-level coursework. Students deemed not yet ready are encouraged or 
required to participate in remedial or developmental coursework before beginning college- 
level courses in those subject areas in which they are found to be academically 
underprepared.! 


Colleges have traditionally used standardized placement tests to determine whether 
students should participate in remediation. Of community colleges surveyed by the National 
Assessment Governing Board in 2010, 100 percent reported using standardized tests for 
math placement purposes, and 94 percent reported using such tests for reading placement 
(Fields & Parsad, 2012). Among four-year institutions, 85 percent reported using 
standardized tests for math placement, and 51 percent reported using such tests for English 
placement (Fields & Parsad, 2012). 


In recent years, however, questions have arisen about the efficacy of standardized 
placement tests as well as the utility of traditional developmental coursework. College 
practitioners and others are concerned about whether too many students are unnecessarily 
required to take developmental education courses before beginning college-level work. The 
courses require students to make a substantial investment of time and money, and many 
students who begin college by taking developmental coursework never complete a college 
credential. Indeed, research shows that the effects of traditional developmental courses are at 
best mixed (Bailey, 2009; Jaggars & Stacey, 2014). 


Evidence also suggests that the use of placement tests alone is inadequate in 
determining which students need remediation (Belfield & Crosta, 2012; Scott-Clayton, 
2012). Partly in response to these findings, colleges are increasingly turning to the use of 


‘Remedial courses are provided to students who are deemed not ready for college-level math or 
English courses or for other courses that depend on college-level reading, writing, or numeracy skills. The 
terms developmental education and remedial education are used interchangeably in this report. 


multiple measures for assessing and placing students (Rutschow & Mayer, 2018). Multiple 
measures placement systems often make use of placement test results but also consider other 
relevant data on incoming students, such as high school grade point average (GPA). While 
research indicates that using multiple measures, and in particular high school GPA, may result 
in the improved placement of students into developmental and college-level courses (Belfield 
& Crosta, 2012; Scott-Clayton, 2012), there is little evidence indicating that using a multiple 
measures placement system influences student outcomes. 


To address this gap, the Center for the Analysis of Postsecondary Research (CAPR) 
initiated an experimental study of multiple measures placement in partnership with the State 
University of New York (SUNY) and seven of its 30 community colleges: Cayuga 
Community College, Jefferson Community College, Niagara Community College, Onondaga 
Community College, Rockland Community College, Schenectady Community College, and 
Westchester Community College. In each setting and for each subject area, math and English, 
a data analytics algorithm was developed — using the college’s own historical student data 
on a number of measures, such as placement test scores and high school GPA — to predict 
the likelihood of success in introductory college-level math and English courses. The 
alternative placement system, which incorporates the newly developed algorithm as well as 
cut points for placement chosen by the faculty, was then used to place incoming students into 
college-level or developmental courses in each subject area. Our study was designed to test 
whether students assessed using the alternative system (the program group) would be placed 
more accurately than students assessed using the status quo system (the control group) and, 
as a result, would be more likely to complete introductory college-level math and English 
courses, persist in college, and earn more college credits — key indicators of likely college 
credential completion. 


The entire study involves three cohorts of students at the seven colleges, those who 
first entered the college intake process in the fall 2016, spring 2017, and fall 2017 terms.” 
Outcomes for each of these cohorts — more than 13,000 students in the full sample — will 
be tracked through the fall 2018 term, resulting in the collection of three to five semesters of 
outcomes data depending on the cohort. The final report on this study will present findings 
on course placement, introductory college-level course completion, credits attempted and 
earned, and persistence. In this interim report, we describe our overall approach to the 


Assignment to program and control groups in the randomized controlled trial occurred just after 
prospective students who began the college intake process were informed about the study, agreed to 
participate, and took a placement test. Some of these study participants (18 percent) did not enroll in any 
course at the college during the same term. For ease of exposition, we refer to all those who chose to 
participate in the study and took a placement test as “students.” We sometimes distinguish them from 
“enrolled students,” the somewhat smaller group of students who took a placement test and then enrolled 
in at least one course. 


evaluation study and discuss how colleges implemented the new placement system, including 
how the data analytics algorithms for each college (one for math and one for English) were 
developed. In addition, we report on first-term impact findings for the first cohort of students 
(who entered the intake process at five of the seven colleges). Finally, we discuss the costs 
involved for these five colleges to set up and use a multiple measures placement system that 
employs a data analytics approach. 


Our initial impact findings are promising. Among a sample of 4,729 students in the 
first cohort, a fifth of math program students and nearly half of English program students 
were placed differently than they would have been under the status quo placement system. 
Most of these students were placed higher than they would have been using placement tests 
alone. In their first semester of college, students in the math and English program groups 
were 3.1 and 12.5 percentage points more likely than control group students to enroll in and 
pass a college-level course in math or English, respectively.* We emphasize that these initial 
findings are based solely on first-term outcomes of the first cohort. The final report on this 
study, which will present longer term evidence on these and other outcomes for all three 
cohorts, will be released in 2019. 


Background and Context 


Developmental Education 


Developmental education is a significant component of public higher education, both 
in terms of student enrollments and in terms of costs. Among 2003—04 beginning 
postsecondary students, 40 percent of those starting at public four-year institutions and 68 
percent of those starting at public two-year institutions took at least one remedial course 
during their enrollment between 2003 and 2009 (Chen, 2016). 


The primary purpose of developmental education is to equip academically 
underprepared students with the skills they need to succeed in college-level coursework. In 
addition, by restricting access to college-level courses to students who meet certain academic 
standards, developmental education requirements may serve the secondary purpose of 
protecting the academic rigor of college-level courses (Bettinger & Long, 2005). 


Studies employing quasi-experimental methods have been used to isolate the causal 
effect of developmental education on student outcomes. The results of these studies vary. 
Bettinger and Long (2005), for example, used instrumental variables to study developmental 


>The rates for control and program math group students were 14.1 percent and 17.2 percent. The 
rates for control and program English group students were 27.2 percent and 39.7 percent. 


education in Ohio’s community colleges and found that remedial education had positive 
effects on college persistence and bachelor’s degree completion. Martorell and McFarlin 
(2011), on the other hand, used longitudinal data from Texas and a regression discontinuity 
design and found that remedial education had little to no effect on the likelihood of earning a 
college degree or on subsequent earnings. 


Jaggars and Stacey (2014) reviewed findings from eight studies that evaluated the 
effectiveness of community college remedial courses across six large systems or states, all 
but one of which used a regression discontinuity approach. These combined studies showed 
that, with some exceptions, developmental education had mostly null and sometimes negative 
impacts on outcomes (such as persistence, passing associated college-level courses, grades in 
college-level courses, credits and credentials earned) for students near the placement score 
cutoffs. They also showed that students placed into lower levels of developmental education 
had a higher proportion of positive effects (five positive vs. six negative and 19 null) than 
students placed in developmental courses who were near the college-level cutoffs (two 
positive vs. 15 negative and 32 null), suggesting that developmental education may have 
differential effects on students depending on their level of academic preparation. 


The overall body of research on the efficacy of developmental education suggests 
that, at best, it does not hurt students, but at worst, it may decrease the likelihood among at 
least some students of attaining their postsecondary education goals (Bailey et al., 2015; 
Bailey, Jeong, & Cho, 2010; Boatman & Long, 2010; Calcagno & Long, 2008; Crisp & 
Delgado, 2014; Melguizo, Bos, Ngo, Mills, & Prather, 2016; Scott-Clayton & Rodriguez, 
2015). Developmental education serves to extend time in college, and long remedial 
sequences can consume students’ financial aid as well as their own resources. These 
consequences can demotivate students, making them less likely to complete their programs 
of study (Bailey, 2009; Crisp & Delgado, 2014; Scott-Clayton & Rodriguez, 2015). In fact, 
only 28 percent of community college students who take a remedial course go on to earn a 
degree within eight years, compared with 43 percent of nonremedial students (Attewell, 
Lavin, Domina, & Levey, 2006). 


The cost of remedial education is high; estimates of the costs to deliver remedial 
courses range from $1.4 billion to nearly $7 billion annually (Long & Boatman, 2013; Scott- 
Clayton, Crosta, & Belfield, 2014). These costs fall directly on students placed into remedial 
courses and indirectly on taxpayers, whose money helps subsidize public postsecondary 
institutions that offer remedial education. As a result, there is both a private benefit and a 
social benefit to ensuring that developmental education is effective, expedient, and offered to 
those most likely to benefit from it. 


Standardized Placement Test Accuracy 


Placement into remedial or college-level courses at most colleges is based on scores 
on a single set of standardized placement tests — most often the ACCUPLACER — in math, 
reading, and writing. These tests do not always assess student skills accurately, and colleges 
that use them may place students into developmental education courses unnecessarily 
(Fulton, 2012). Placement test scores are not highly correlated with success in initial college- 
level courses: Doing poorly on a placement test does not reliably indicate that a student would 
be unsuccessful in a college-level course. As a result, using test scores for placement leads to 
placement errors for large numbers of incoming students (Bailey et al., 2015; Belfield & 
Crosta, 2012; Hodara & Cox, 2016; Scott-Clayton et al., 2014). 


Scott-Clayton (2012) identified large predicted “severe error rates” associated with 
placing students using standardized placement tests alone. A severe error rate refers to placing 
students in remediation who would be expected to receive a grade of B or better in college- 
level courses (underplacement) or placing students in college-level courses who would be 
expected to fail (overplacement). While both types of errors should be mitigated, Scott- 
Clayton’s research suggests that the occurrence of underplacement far exceeds the occurrence 
of overplacement. Using student data from a large urban community system, she found 
predicted severe overplacement rates of about 6 and 5 percent in math and English but severe 
underplacement rates of about 18 and 29 percent in the respective subject areas. Scott-Clayton 
further established that these severe error rates could be reduced by employing multiple 
measures for placement. In particular, the high school GPA was found to be a strong predictor 
of success in college-level courses. 


Approaches to Multiple Measures Placement 


Varied measures, used alone or in combination, can be employed to place students into 
developmental and college-level courses. In addition to standardized placement test scores, 
some measures that are in current use are GPAs and other information from high school 
transcripts, scores on writing assessments, noncognitive tests measuring psychosocial 
characteristics, and student self-assessments. Varying levels of evidence support the use of each 
of these measures, with some more thoroughly studied than others (Barnett & Reddy, 2017). 


An increasing number of colleges are exploring or beginning to use multiple measures 
in placement decisions. In a survey conducted in 2016, 57 and 51 percent of community 
colleges reported using multiple measures for placement in math and English, whereas only 
27 and 19 percent reported having done so in 2011 (Rutschow & Mayer, 2018). Colleges 
using multiple measures have employed a variety of methods to combine particular measures 
in order to place students more accurately. The simplest of these is a waiver system, in which 
one or more criteria can be used to exempt students from developmental education 


requirements. Another method involves the use of decision bands; students with placement 
test scores within a certain range are further evaluated using measures such as high school 
GPA or the score on a noncognitive test to further determine placement. Alternatively, 
historical student performance data from a college can be analyzed to weight various 
measures of student assessment and achievement in a way that best predicts future outcomes, 
the method used in the current research. Algorithms reflecting these weights, along with 
chosen cut points, can then be used to place students (Barnett & Reddy, 2017). 


To be useful in real-world settings, placement instruments and methods must balance 
accuracy with cost-efficiency. For example, scored personal essays and in-person advising 
meetings that leverage faculty experience can improve the accuracy of placement and 
increase success rates for students (Duffy, Schott, Beaver, & Park, 2014). However, 
undertaking these activities is much more resource-intensive than using traditional placement 
tests, which are largely automated and more easily scaled (Hodara, Jaggars, & Karp, 2012). 


Effectiveness of Multiple Measures Placement 


Studies show that multiple measures placement methods that incorporate high school 
information, and in particular high school GPA, can significantly improve placement at a 
relatively low cost (Hodara et al., 2012; Belfield & Crosta, 2012).* Studies by Scott-Clayton 
(2012) and Belfield and Crosta (2012) found that high school GPA can help predict college 
performance and could be used to place students more accurately than scores on placement 
tests alone. Both studies suggest that an optimal placement strategy would take into account 
both high school transcript data and placement test scores. 


Results from a small randomized experiment at a Midwestern community college 
(Marwick, 2004) showed that students placed using either one of two multiple measures 
approaches were more likely to take and succeed in higher level math courses than were 
students placed using standardized test scores alone. One method incorporated placement test 
scores and performance in high school math; the other method involved an advisor-mediated 
student choice scenario in which test scores, high school preparation, and other factors were 
discussed in an advising session. While this study was very small, the results suggested that 
further evaluation of multiple measures placement is warranted. 


Statewide changes in placement policies are allowing for broader examinations of 
alternative placement methods. North Carolina instituted a statewide reform that began in 
2013 and was required to be used by all colleges by fall 2016. The policy exempts students 


‘Belfield and Crosta (2012) found that including additional information from the high school 
transcript (e.g., the number of courses taken in math or English, or the total number of high school credits) 
to predictive models that already included high school GPA contributed little to no additional value. 


from remediation based on certain criteria. For example, students who graduated from high 
school within five years with a GPA of 2.6 or above are exempted from remediation. If the 
GPA threshold is not met, colleges also grant exemptions based on SAT or ACT scores. Only 
those students who do not meet the GPA or SAT/ACT requirement must take a placement 
test (Dadgar, Collins, & Schaefer, 2015). 


In California, the 2017 passage of Assembly Bill 705 called for all community 
colleges in the state to modify their placement practices so that high school data is used as a 
primary measure of college readiness by spring 2019. While system-wide changes are 
underway, individual community colleges have already begun to implement multiple 
measures placement systems. Before passage of the bill, Long Beach City College developed 
an algorithm that uses student high school achievement in addition to standardized placement 
test scores to assess students. The algorithm weights each measure on the extent to which it 
predicts student performance in college courses (Long Beach City College, Office of 
Institutional Effectiveness, 2013). Using the multiple measures algorithm increased student 
placement into college-level courses from 15 to 60 percent in English and from 10 to 50 
percent in math, with no significant change in student success rates (Dadgar, et al., 2015). 
Many other California colleges are now implementing versions of this approach, which is 
similar to the one undertaken in the current project. 


About CAPR 


Established in 2014, the Center for the Analysis of Postsecondary Readiness (CAPR) 
is a partnership of research scholars supported by the Institute of Education Sciences, U.S. 
Department of Education, and led by the Community College Research Center (CCRC) at 
Teachers College, Columbia University, and MDRC, a nonprofit research and development 
organization. In addition to the study described here, CAPR is conducting two additional 
major studies, one based largely on a nationally representative survey that aims to provide a 
comprehensive understanding of the landscape of developmental education and reform in 
two- and four-year colleges across the country, and one that evaluates an alternative model 
of developmental math programming that shortens students’ time in remediation, tailors 
content to students’ academic paths, and uses student-centered instruction. CAPR also carries 
out leadership and outreach activities aimed at improving college readiness. 


Chapter 2 
Placement System and Study Design 


The current study uses a randomized controlled trial to compare the effects on student 
outcomes of placing students into developmental or college-level courses with either a 
multiple measures, data analytics placement system or a status quo system that uses just one 
measure, placement test scores. In order to carry out this evaluation, an alternative placement 
system had to be created and implemented, and random assignment procedures had to be 
established. Researchers and personnel at each college collaborated in these activities. We 
describe the approach used as well as the broader study design in this chapter. 


There are five research questions guiding the study: 


1. How isa multiple measures, data analytics placement system implemented, 
taking into account different college contexts? What conditions facilitate 
or hinder its implementation? 


2. What effect does using this alternative placement system have on 
students’ placements? 


3. With respect to academic outcomes, what are the effects of placing 
students into courses using the alternative system compared with 
traditional procedures? 


4. Do these effects vary across different subpopulations of students or by 
college? 


5. What are the costs associated with using the alternative placement system? 
Is it cost-effective? 


To answer Question 1, we conducted two rounds of implementation site visits to each 
of the seven colleges; we spoke with key personnel, including administrators, staff, and 
faculty. To answer Questions 2 through 4, this study tracks eligible students who first began 
the intake process at a participating college in the fall 2016, spring 2017, or fall 2017 term 
through the fall 2018 term. These students were randomly assigned to either the program 
group or the control group. The study design calls for impact analyses to be performed twice 
— once early in the study, following the end of the first cohort’s first semester, and again for 
all three cohorts following the conclusion of the study’s tracking period. 


For the first set of analyses, which are presented in this report, student data were 
collected in early 2017 from the five colleges that began participation in the study in fall 
2016, as well as from the SUNY central institutional research unit. Student outcomes data for 


all three cohorts will be collected from the colleges and SUNY during the spring of 2019 for 
the second and final set of analyses, which will allow researchers to observe students’ 
outcomes (see Appendix Table A.1) for three to five semesters following placement, 
depending on the cohort. 


To answer Question 5, we are carrying out a cost-effectiveness analysis that will 
incorporate data collected at the end of the project in spring 2019. Chapter 5 of the current 
report presents a cost-only analysis on the five colleges that began enrolling participating 
students in fall 2016. The current report also presents implementation findings (Chapter 3) 
and early impacts findings on the first cohort of students (Chapter 4). The final report on the 
results of this study will be released in 2019. 


Site Descriptions 


Seven SUNY colleges are participating in this study. Many had a prior interest in 
assessing the effectiveness of their existing placement system before they got involved, while 
others saw participation as an opportunity to improve knowledge and practices in student 
placement. The colleges are diverse in terms of size and population served (see Appendix 
Table A.2). While the smallest of the colleges serves roughly 5,500 students, the largest 
serves over 22,000 students annually. As is common in community college settings, a large 
portion of students at the colleges attend part-time, and many are adult learners, with between 
21 and 30 percent of students over the age of 25. Most of the colleges serve large numbers of 
students who receive financial aid — more than 90 percent of students receive financial aid 
at five of the seven colleges. The colleges have transfer-out rates of between 18 and 22 
percent; their three-year graduation rates are between 15 and 29 percent. 


All of the colleges have an open-door admissions policy, meaning that they do not 
have entry requirements for incoming students beyond having graduated from high school or 
earned a GED. The colleges tend to serve local student populations, and most have 
relationships with their region’s high schools both for offering dual enrollment programs and 
to facilitate the admissions process from high school to college. Each college has a small 
population of students who live on campus or who moved to attend the college. 


The colleges offer a wide selection of programs of study, including a few that 
particular colleges have developed and gained a strong reputation for, such as nursing, 
electronic communications, culinary, and music programs. Further, each college has varying 
on-campus initiatives that reflect the goals and priorities of the college. For example, one 
college has made a big push to increase the diversity of its faculty to better match the student 
population it serves. Another college has established an academic success center and has 
taken part in the START-UP NY program to foster private/public partnerships. And 
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especially germane to this study, one college has designed programs called Prep for Success 
and Math Boosters to help students brush up on their skills and then retest if they are not 
initially placed in college-level courses. 


Creating a Data Analytics Placement System 


Given the differences among the colleges, such as the different student populations 
they draw from, the data analytics algorithms employed to assess program group students 
were created for each college individually, using historical data on previous students at each 
college. The resulting algorithms and historical data also allowed us to estimate historical 
misplacement rates at each college (see Box 2.1). After the math and English algorithms were 
developed, faculty at each college chose cut points on the range of scores for each algorithm 
that were then used to place program group students into developmental or college-level math 
and English.° 


Using Historical Data to Develop Algorithms 


Historical high school and placement test data were needed to create predictive 
algorithms at each college. Five colleges in the study had been using ACCUPLACER tests 
for several years. A sixth college had been using ACCUPLACER tests for English but had 
transitioned from a homegrown math assessment to the ACCUPLACER set of math tests 
more recently; this college is therefore testing the use of the alternative placement system for 
English placement only in this study. The seventh college in our sample had been using 
COMPASS tests, standardized placement tests which were discontinued by the provider 
(ACT) shortly after this study began. This college is also testing the use of the alternative 
system for English placement only. At this college, the predictive algorithm that is being 
tested in the alternative placement system does not make use of any placement test scores; 
rather, it is based only on high school GPA and other high school data. The status quo 
placement system in this case uses only scores from ACCUPLACER, the test that the college 
selected to replace the COMPASS. 


CAPR researchers worked with the appropriate personnel at each college as well as 
SUNY’s central institutional research unit to obtain historical data on students who first 
enrolled during the 2011-12, 2012-13, and 2013-14 academic years. Data on multiple 
measures, such as high school performance and placement test scores, as well as data on 
outcomes in college-level courses were used to create algorithms for predicting student 


5The colleges often used multiple cut points on the range of each algorithm’s student scores to 
place students into different levels of developmental coursework and different levels of college-level 
coursework in math and English. For this study, however, we are considering only two placement 
alternatives: developmental versus college-level placement. 
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performance in college-level math and English among students in the study sample. In some 
instances, data on these measures were available in college systems, stored in digital format. 
Other colleges maintained records of high school transcripts as digital images; in these cases, 
the needed data had to be entered into computer systems by hand. 


In order to estimate the relationships between the measures, or “predictors,” in the 
dataset and performance in an initial college-level course, the historical data used for analyses 
were restricted to students who took placement tests and enrolled in a college-level course 
without first having taken a developmental course. This set of students constituted our 
estimation sample. We then regressed success in a college-level course on various sets of 
predictors using a linear probability model.° (Alternative models are described by Hastie, 
Tibshirani, and Friedman [2009], but more intricate models we tested yielded similar results.) 


For each of the colleges, we began by creating a model for estimating the relationship 
between high school GPA and success (defined as earning a grade of C or higher) in an initial 
college-level course in a given subject, math or English (see Equation 1 below). We then 
estimated the relationship between placement test scores and success in these initial college- 
level courses (Equation 2). A third model included both high school GPA and placement test 
scores for the appropriate subject (Equation 3). A fourth model added additional information 
where such information was available (Equation 4). Added variables include the number of 
years that had passed since high school completion and whether the student’s diploma was a 
standard high school diploma or a GED, SAT scores, ACT scores, and scores on the New 
York State Regents Exams where they were available (see Appendix Tables A.3 and A.4), as 
well as interaction terms and nonlinear terms for certain variables. Identical procedures were 
followed for both math and English. 


(1) Pr(C or Better) = a+ (HS GPA)B, + € 
(2) Pr(C or Better) = a+ (ACCUPLACER)B, + € 
(3) Pr(C or Better) = a + (HS GPA)B, + (ACCUPLACER)B, + € 


(4) Pr(C or Better) = a + (HS GPA)B, + (ACCUPLACER)B, + XB3 + € 


Each variable in the model was accompanied by a corresponding missing indicator. Missing 
indicators were entered into the model equations as | if a student was missing that data point and zero if 
the student had a value for that data point. 


12 


While researchers may look at the individual covariates in a traditional study, the 
focus of this analysis is the overall predictive power of each model. We therefore used the 
Akaike Information Criterion (AIC) to compare the models. The AIC is a measure of model 
fit that combines a model’s log-likelihood with the number of parameters included in a model 
(Akaike, 1998; Burnham & Anderson, 2002; Mazerolle, 2004). When comparing models, a 
lower AIC statistic indicates a better fitting model (Mazerolle, 2004). The best fitting model 
was the one selected for use at each college in the study. Appendix Tables A.3 and A.4 list 
the full set of variables used in each college’s algorithm for math and English. 


Estimation of Historical Misplacement Rates at Each College 


The data analytics algorithm that was created for each college (in each subject area) 
also allowed us to compute historical underplacement and overplacement rates for math and 
English. We define an underplaced student as one placed into a developmental course who 
could have succeeded in an initial college-level course in the same subject area by earning a 
grade of C or higher.’ In conducting analysis on underplacement, a student’s probability of 
succeeding in the college-level course is calculated using the parameters estimated by each 
college’s best fitting model. We define an overplaced student as one unable to pass a college- 
level course who was nonetheless placed into such a course. Importantly, this is not simply 
the inverse of passing with a C or higher, since a D is not considered a failing grade. 
Nonetheless, the model for overplacement uses the same set of predictors selected in 
modeling underplacement. For example, if Equation 4 from above is selected as a college’s 
best fitting model, then each student’s likelihood of failing the initial college-level course is 
calculated using the following equation: 


(5) Pr(Fail) = « + (HS GPA), + (ACCUPLACER)$, + XB3 + € 


The overplacement and underplacement rates for each college are simply averages of 
these individual probabilities. In keeping with techniques introduced by Scott-Clayton (2012), 
we sum the overplacement rate and the underplacement rate to generate a total error rate. 


Appendix Table A.5 shows the mean estimated underplacement, overplacement, and 
total error rates for each of the five colleges. The results indicate that placement accuracy is 
an issue in both math and English for the five colleges in this phase of the study. The 


7Scott-Clayton (2012), Belfield and Crosta (2012), and Scott-Clayton et al. (2014) used a passing 
grade of B or better as the outcome of interest, arguing that this higher threshold ensures that only those 
who are “severely” underplaced will be identified by the model. Given our threshold of a grade of C or 
better, we distinguish our error rates from the rates generated in those prior studies. 
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proportion of misplaced students ranged from 32 to 50 percent in math and from 43 to 52 
percent in English. The error rates were higher in English than in math at three colleges, and 
very similar to one another at one college. A fifth college had higher error rates in math than 
in English. 


Prior research on first-time entrants in a large urban community college system 
(Scott-Clayton, 2012) suggests that underplacement is typically a larger problem than 
overplacement. Our results on historical misplacement at these five colleges are consistent 
with these findings. With one exception (math misplacement rates at one college), rates of 
underplacement were higher than rates of overplacement for both math and English at each 
of the colleges, and in most cases much higher. 


Choosing Cut Points for Projected Placement and Pass Rates 


After data analytics algorithms were established at each college, we used the 
coefficients from the regressions to simulate placement and success rates as a basis for faculty 
decisions on where to establish cut points that distinguish students ready for college-level 
courses from those needing remediation. Consider the following simplified example using 
Equation 3 from above. Let Y represent the predicted probability of success in a college-level 
course. We can use regression coefficients and a student’s own placement test scores and 
high school GPA to predict the probability of earning a C or better in college-level math (Y) 
for any new student 7. A set of decision rules can then be determined based on these predicted 
probabilities. If the college has one level of developmental math placement and one college- 
level course placement, the decision rule may be: 


College level if ¥, > 0.6 


Placement; = es 
Developmental if ¥; < 0.6 


For each college, we generated spreadsheets projecting the share of students that 
would place into a college-level course at any given cut point on Y, as well as the share of 
those students we would anticipate earning a C or better in that course. These spreadsheets 
were given to colleges so that faculty in the relevant departments could set cut points for 
students taking math or English courses. 


Table 2.1 shows a hypothetical example of one such spreadsheet provided to colleges. 
The top panel shows projected math placement statistics, and the bottom panel shows 
projected placement statistics for English. The first column shows the cut point, or the 
minimum allowable probability of success for students, that produces the projected share of 
college-level placements (second column) and pass rates (conditional on college-level 
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placement; third column). The top, highlighted row in each panel shows the historical 
placement and pass rates at the college. 


As an example, the historical placement rate for math in the table is 30 percent. The 
third column shows the pass rate, based on the receipt of a grade of C or higher, in the initial 
(gatekeeper) college-level course. The historical pass rate for math in this example is 50 
percent, conditional on placement into the college-level math course. 


Table 2.1 


Hypothetical Spreadsheet on Projected College-Level Placement and Completion Rates 
(With Grade C or Higher) at Given Cut Points 


Math Success 


Cut Point Percent Who Will Pass 
(Minimum Probability Percent Who Will Place College-Level Course 
of Success) Into College-Level Course With Grade C or Higher 
Historical 30% 50% 
45% 40% 60% 
55% 20% 70% 
65% 10% 715% 


English Success 


Cut Point Percent Who Will Pass 
(Minimum Probability Percent Who Will Place College-Level Course 
of Success) Into College-Level Course With Grade C or Higher 
Historical 40% 60% 
45% 75% 60% 
55% 60% 65% 
65% 20% 70% 


Below each highlighted row is shown what would happen to placement and pass rates 
at different cut points chosen for scores on the algorithm. For math, the first cut point shown 
is 45 percent, which means that to be placed into college-level math under the algorithm, a 
student must have a predicted probability of receiving a C or higher in the gatekeeper math 
course of at least 45 percent. If this 45 percent cut point were used, Columns 2 and 3 show 
what share of students would be placed into college-level math under the algorithm (Column 
2) and what share are projected to pass this course conditional on placement (Column 3). In 
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this example, for math, if the 45 percent cut point were used, the algorithm would place 40 
percent of students into college-level math, and 60 percent of those students would be 
projected to pass the course with a C or higher. The cut point differs from the projected pass 
rate. The cut point represents the /owest probability of passing for any given student; the cut 
point implies that every student must have that probability of passing or higher.® 


Many faculty opted to create placement rules that either (1) kept pass rates in college- 
level courses similar to historical pass rates or (2) kept college-level placement rates similar 
to historical placement rates. Under the first approach, the algorithm tended to predict 
increases in the number of students placed into college-level coursework. For instance, in the 
example shown in Table 2.1, the historical pass rate for college-level English is 60 percent. 
A cut point of 45 percent would induce the same pass rate, 60 percent, but would place 75 
percent of students into the college-level English course. 


Implementing the Alternative Placement System 


Colleges in the study had two options for implementing the data analytics placement 
system. At colleges running the system through ACCUPLACER, researchers programmed 
custom rules into the ACCUPLACER software for students selected to be part of the program 
group. The rules specified the ACCUPLACER placement determination for every 
combination of multiple measure values used in the algorithm, which were accessed from a 
pre-registration file created and uploaded with data for each incoming student. 


Other colleges conducted their placement through MDRC’s custom-built server and 
therefore did not need to create a pre-registration file. Instead, student information was sent 
to MDRC servers in one of two ways. Either all information was uploaded together and a 
placement decision was returned for each student, or students’ supplemental information was 
uploaded in batches and test scores were uploaded individually by counselors after students 
completed their testing. The values of the uploaded multiple measures and test scores were 
then multiplied by their respective algorithm weights and summed to generate the predicted 
probability of success and the corresponding placement, which was returned to the college. 


8For instance, if the cut point were 40 percent, then every student placed into the college-level 
course would need to have a 40 percent chance or greater of passing the college-level course — most 
students would have above a 40 percent chance. This means we should expect the projected pass rate to be 
higher than the cut point. If higher cut points are used — meaning that students must have higher 
probabilities of passing in order to be placed into the college-level course — then the share placed into the 
college-level course declines but the anticipated pass rate increases because the standard for placement 
becomes more challenging. 
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Randomized Controlled Trial Procedures 


The research design for this study was selected to meet What Works Clearinghouse” 
evidence standards without reservations. Our procedures were as follows. First, entering 
prospective first-year students arrived at each college for the intake process. Those with 
waivers based on SAT scores or with other exemptions from both math and English 
placement testing were not placement tested at all but rather went straight into college-level 
courses; they were not part of the study. Before taking placement tests, the remaining students 
(some of whom took tests in only one subject area, math or English'®) were informed about 
the research, afforded the opportunity to seek additional information, and were able to opt 
out if they wished.'! Those who continued took placement tests and were randomly assigned 
to be placed using either the status quo method (control group students) or the method using a 
multiple measures, data analytics algorithm (program group students). After taking placement 
tests, students were notified of their placements into developmental or college-level courses 
either by a college staff member or through an online portal, depending on the college. It is 
important to recognize that nearly one fifth of students who were randomly assigned to the 
control or program group and who took a placement test later decided not to enroll in any 
course in the fall 2016 term. We nonetheless include such persons as “students” for purposes 
of intention-to-treat analysis and sometimes distinguish “students” from “enrolled students,” 
those who did enroll in at least one course at the college in the fall 2016 term. 


The random assignment process was integrated into the existing placement 
procedures at each college, though the way that this was accomplished was tailored to 
individual campuses. Irrespective of the randomization mechanism, control group students 
followed status quo placement procedures, and program group students were placed using 
the alternative placement system. Students did not receive information on which group they 
were assigned to. 


Established by the Institute of Education Sciences, the What Works Clearinghouse reviews 
research on different programs, products, practices, and policies in education to provide educators with 
information needed to make evidence-based decisions. 

‘0Students who took a placement test only in math were not considered in the analysis of English 
outcomes, and students who took a placement test only in English were not considered in the analysis of 
math outcomes. 

"Students who opted out never entered our study; thus, we cannot report on the exact number 
of them. 
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Chapter 3 
Implementation Findings 


In addition to a randomized controlled trial conducted to measure impacts of 
alternative placement on student outcomes, this study includes an examination of the 
processes required to implement a multiple measures, data analytics placement system and 
considers the factors that can hinder and facilitate implementation. To carry out this 
examination, CAPR research teams visited each of the seven participating colleges on two 
separate occasions. An additional set of visits took place early — during the winter of 2015 
— to provide information to college personnel about the project design and to discuss 
essential elements of implementation procedures; these visits were not used to collect 
implementation data. Subsequent visits occurred during the summer of 2016, by which point 
most colleges had begun enrolling students in the study. The final round of site visits took 
place in the spring of 2017. These visits were primarily designed to collect data about 
implementation, but they also served as opportunities for researchers and college personnel 
to discuss and troubleshoot any problems with implementation. The personnel who 
participated in interviews and focus groups during the two rounds of site visits included 
representatives from college administration, admissions, testing, advising, information 
technology/institutional research (IT/IR), registrars, and faculty. 


Interviews and focus groups were conducted using semi-structured interview guides. 
With the consent of participants, these sessions were tape-recorded. Researchers took detailed 
notes when participants did not consent to recording. Recordings were transcribed and 
supplemented with researchers’ notes for the purpose of analysis. Transcripts were loaded 
into the qualitative analysis platform Dedoose for coding and analysis. Although formal 
measures of inter-rater reliability were not calculated, members of the research team worked 
to align their applications of coding and met at regular intervals to discuss any points of 
uncertainty. The coding process allowed researchers to identify themes that colleges shared 
and ways in which colleges differed in their experiences. These results are presented below. 


Status Quo Placement Procedures 


The seven colleges in this study all followed very similar placement procedures 
before beginning their involvement with this project. Substantive differences among them 
were confined to the number of levels in each college’s developmental math and English 
programs,” the subset of ACCUPLACER tests used for assessment, and the cut scores on 


In this study, we distinguish only two kinds of placements, developmental and college-level 
placements. 
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the tests chosen for placement into different courses. Under the status quo system, after 
students apply to each college, their files are reviewed to determine which tests, if any, they 
must take. Some students are exempt from placement testing; exemptions are typically based 
on scores on the Regents exam, AP exam, or SAT. Students who are not exempted either 
schedule a testing session or visit the testing center during designated drop-in hours to take 
placement tests. Subsequently, students meet with a counselor or advisor, who discusses the 
placement results and assists the students with initial course registration. In most instances, 
high school transcripts are not required in order to complete this process, though colleges do 
obtain proof of high school completion prior to a student’s actual enrollment. 


Strengths 


During our site visits, discussions with college personnel about the status quo 
placement system often underscored that the system had weaknesses in placing students 
accurately (discussed below). However, stakeholders did note a few strengths of the system. 
One was the straightforward nature of comparing a student’s score on a test with an 
established cut score. This method was easy for students, counselors, and faculty to 
understand, especially when compared with the relative opacity of the data analytics system 
used for program students in the study. 


Related to the system’s simplicity was its efficiency. Students could be placed into 
coursework very quickly, without the need to obtain additional information from a high school 
transcript. This was helpful when college personnel were required to process a high volume 
of admissions paperwork, and especially when students arrived shortly before the start of a 
semester. Indeed, one participating college needed to suspend study procedures for the first 
week of each fall semester in order to make sure that students who arrived that week could 
start their courses immediately rather than waiting a day for an algorithm-based placement. 


Efficiency was also important in terms of the student experience. One administrator 
emphasized the value of having a student leave the premises with a schedule in hand after 
taking the placement tests. He believed that this increased the likelihood that the student would 
actually enroll in college in the fall, even if the course placement was not necessarily accurate. 


Furthermore, many faculty were comfortable with existing tests or liked the idea of 
being able to rapidly learn about the ability levels of students in a given course based on a 
review of their test scores. A few faculty also felt that the status quo system was especially 
useful in placing adult learners who had been out of school for several years and whose high 
school records might not accurately reflect their readiness for college. For example, one told 
us, “Adult learners, who had been away from school for a while.... If they did well on 
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ACCUPLACER, it indicated to us that there was a shift in what may have been on their 
previous academic record.” 


Weaknesses 


Among the most frequently expressed concerns with the status quo system was the 
way in which students approached the placement tests. Staff and faculty often observed that 
students rarely prepared for the tests. Most attributed this to students not knowing how 
important the tests were, a perspective that may have been reinforced by messaging from the 
colleges. In some cases, personnel noted that students arriving for testing were told, ““We just 
want to see where you are to make sure we put you in the [right] courses. You’re coming to 
[the college] no matter what. Don’t even worry about this.” 


Others noted that students focused on getting through the tests as quickly as possible 
rather than taking care with their answers. Taking multiple tests in one day may have 
contributed to this impulse on the part of incoming students. One math faculty member told 
us, “I do know that we heard that in math at the time that, ‘Oh, that was the third test I had to 
take.” Finally, several interviewees believed that some students were not accustomed to 
taking tests on a computer. This concern was particularly acute in the case of older students. 


The interviewees frequently reiterated their belief that the tests were not doing a good 
job of placing students into the appropriate level of coursework. Many college personnel felt 
that the tests did not properly assess student skills, and many noted that the CAPR estimates 
of placement errors using historical data (see Box 2.1), provided to them early in the study, 
confirmed their suspicions. Math and English faculty offered slightly different perspectives 
about this. 


Math faculty often noted that the placement tests might be fine for identifying 
students at the top and the bottom of the distribution, but not for the middle of the distribution, 
which they considered to be much tougher to gauge. These faculty also voiced concerns about 
student mastery of different kinds of math. Some noted instances when students might place 
into college-level courses based on their algebra subtest score, though their arithmetic subtest 
score indicated a need for developmental coursework." 


English faculty emphasized misalignment between the skills measured by the test and 
the skills required to successfully complete a first-year college-level English course. 
Describing the ACCUPLACER sentence skills subtest, one faculty member told us: 


The tests are computer adaptive at each college, but not always in equivalent ways; at some 
colleges, students who score poorly on the arithmetic subtest do not take the algebra subtest. 
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I would say the ACCUPLACER isn’t a writing test. It’s more like 
... a multiple choice test that asks you to complete sentences. You 
know, “What would be the best completion of this sentence?” 
Because ... it’s not an actual writing test, some people don’t test 
well. I would say that’s also a drawback to it. 


Most colleges participating in this study used the computer-graded WritePlacer, an 
ACCUPLACER subtest administered at some colleges, but one faculty member noted that the 
computer simply measured “how few errors somebody makes and how long the paragraphs 
... and sentences are,” which did not provide an accurate portrait of a student’s skills. A few 
colleges did have a hand-graded writing assessment, either as the primary assessment or as a 
method for students to appeal a placement decision. A faculty member at one of these colleges 
noted, however, that administering this assessment was both time- and labor-intensive. 


A final concern, related by a handful of participants, involved the cost of 
ACCUPLACER to the college. Though some stated that the cost of each test was not that 
high, others noted that the aggregate cost of the ACCUPLACER contract was significant and 
wondered whether the college might devise an effective system that did not require this 
expenditure. 


Implementation of the Data Analytics Placement System 


Overall, implementation of the multiple measures, data analytics placement system 
created a significant amount of up-front work to develop new processes and procedures that, 
once in place, generally ran smoothly and with few problems. At the beginning of the project, 
colleges underwent a planning process of a year or more, in close collaboration with the 
research team, in order to make all of the changes required to begin implementing the 
alternative placement system. Each college took the following steps. (Items in italics were 
only required because of involvement in the research.) 


e Organized a group of people to take responsibility for developing the new 
system and designated an overall project lead. 


e Compiled a historical dataset in order to create the college’s algorithms 
(one each for math and English) and to conduct related analyses. 


e Developed or improved processes for obtaining high school transcripts 
for incoming students. 


e Increased the number of students for whom high school data points were 
entered into the college computer system before students went for testing. 
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e Created procedures for uploading high school data into a data system 
where it could be combined with test data at the appropriate time. 


e Modified admissions procedures to inform students of the research and 
give them an opportunity to opt out. 


e Changed testing procedures and data management systems to permit 
placement services to be provided to both program and control group 
students. 


e Changed IT systems to capture the placements derived from the use of 
multiple measures. 


e Created new placement reports for use by students and advisors. 


e Provided training to testing staff and counselors on how to interpret the 
new placements and communicate with students about them. 


e Conducted trial runs of the new processes to troubleshoot and avoid 
problems during actual implementation. 


e Offered opportunities for the college community to learn about the new 
system and the research. 


Impact of the Alternative System on Various College Groups 


The implementation experiences of different groups on campus varied, with some 
more directly affected by changes in the placement system than others, particularly at 
different stages of the process. Typically, IT staff, admissions and testing staff, registrars, and 
college administrators were involved intensively early on, while advising staff and faculty 
were more likely to be involved later. 


Admissions 


Implementation of the alternative placement system generally required admissions 
offices to become more proactive about collecting student high school transcript data because 
availability of the high school GPA was central to the new system. Experiences varied from 
college to college. Those that already required students to submit their transcripts in order to 
enroll were not heavily affected. Yet most of the colleges did not previously include obtaining 
high school transcripts as a standard part of their workflow, and tracking them down and 
entering associated data into college computer systems proved to be time-consuming. In some 
cases, college representatives had to collaborate closely with high schools to obtain transcript 
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data more regularly and rapidly for incoming students. Collecting transcript data was 
particularly difficult when student applicants did not apply immediately following high 
school. In one case, admissions staff estimated that transcript collection and data entry 
doubled the office’s workload. 


Information Technology 


In many cases, college IT departments took on the bulk of the work required to set up 
the alternative placement system. They needed to connect computer systems and data across 
different offices and interfaces, including registration files from admissions, test results from 
ACCUPLACER, and data on what placement determinations were made through the 
algorithms. In addition, IT personnel addressed problems that arose as staff in different 
departments began using the new procedures. Overall, IT staff reported that there was a lot 
of up-front work to create more automated processes that could save time later once the 
system was established. This up-front work took an estimated two to six weeks to accomplish. 
Once the system was in place and had been tested, it generally ran smoothly. 


Registrars 


Implementing the alternative placement system generally meant that course 
prerequisites had to be changed in each college’s registration system. Registrars reported 
spending time to make sure that the new placement designations were reflected in college 
computer systems before registration took place for the affected semesters. With sufficient 
advanced notice, registrars reported that this process was manageable. 


Testing 


Like the admissions departments, testing staff told us that changing the placement 
system caused an initial increase in workload, as they had to create a new process to link 
testing data to other data required by the multiple measures system. Once this was done, 
however, the additional increase in workload was minimal. Testing staff also had to come up 
with new procedures for dealing with students who needed to retest and for “one-stop” 
students — those who go through the entire admissions and enrollment process in a single 
day. With regard to retesting, it was less clear whether it would benefit students to take the 
test again under the alternative system, given that much of their placement determination for 
program group students was based on their high school information. Further, the new 
procedures sometimes made same-day placements infeasible; instead, it often took more than 
a day to have all needed data to make a placement determination. This could be inconvenient 
for testing staff and for students. 
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Counseling/Advising 


Staff had to gain comfort with an advising system that involved looking at placement 
results rather than raw test scores. This did not significantly change the workload for 
counseling/advising staff. However, some reported feeling torn about what to do when they 
were able to see both raw scores and placement results, particularly if they felt that following 
one or the other would benefit the student, for example, by placing them into a higher level 
course. Colleges fixed this problem by removing the raw scores from reports so that advisors 
could simply follow the placement results. After making this adjustment, most staff reported 
that it was easy to forget that the study was even going on as they simply followed the 
placement decisions on the reports. 


Faculty 


Overall, there was no consistently reported impact of changes on the faculty 
experience or approaches to instruction resulting from use of the alternative placement 
system. In many cases, faculty reported that nothing noticeable changed. In other cases, 
faculty described ways that the new placement system affected their work. A few college- 
level math and English instructors reported observing students with a broader range of skills 
and abilities in their courses, while others perceived an overall decrease in student skills and 
abilities. Developmental education instructors in some cases reported that their classrooms 
became more homogenous. In cases where faculty did notice a change in student skills and 
abilities, they mentioned having to adjust their instruction in order to meet the different needs 
of students. 


Students 


According to college faculty and staff, the alternative placement system did not 
substantially affect students’ matriculation experiences, as the process for applying and 
enrolling generally stayed the same. Furthermore, students were not told whether they were 
in the program or control group. The only noticeable change to the student experience at some 
colleges was a delay between testing and receiving placement results. In these cases, students 
would have previously received their placement immediately after completing their tests. 
Under the alternative system, some colleges built in a one-day delay to allow time for a batch 
upload of data each night. 


Students also no longer saw a test score, and this led to some confusion about how 
their placement had been determined and whether retesting was advisable. Overall, however, 
college personnel reported that the student experience did not change very much. 
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Challenges of Alternative Placement 


For the most part, college representatives believed that it was important to change 
their placement strategies. However, they were also clear about the challenges involved in 
undertaking this initiative. Some of the challenges mentioned had to do with the process of 
engaging in reform generally, while others pertained to the multiple measures, data analytics 
approach to placement. 


Engagement With Stakeholders 


Personnel at some colleges felt that inadequate time had been spent discussing and 
understanding the alternative placement system. In some cases, this was because of the 
timeline imposed by the research, which aimed to begin student intake in the fall of 2016. In 
other cases, there were groups of people at the college who were not initially consulted. This 
may have been because they were originally viewed as less likely to be affected by changes 
to the placement system, or simply because of a desire to keep the process moving quickly. 


I think that’s one of the key things that probably came out of all of 
this for all of us: to know any kind of changes that we were planning 
to do with placement testing in general, you’d have to be planning 
so much further out. You have to get these processes started so 
much sooner in order to really address it adequately. 


College cultures varied in the way that change was perceived, and in levels of trust 
among groups within the college. At some colleges, there was an eagerness to try out new 
and promising practices. At other colleges, there appeared to be a history of mistrust of 
change that was perceived to be imposed by the college administration or by outside groups 
such as CAPR. 


Use of High School Transcript Data 


While some colleges had always required students to bring in high school transcripts 
at the time of admission, others had not. High school transcripts that were obtained were 
habitually scanned in pdf format and saved by all of the colleges. However, only limited 
numbers of colleges routinely entered data from the transcripts into the college’s computer 
system, permitting their use in a data analytics placement system. To participate in this 
project, additional data entry was often needed. Yet many college leaders also acknowledged 
that these data were important to incorporate into the college’s computer system for use in a 
range of analyses. 


Finally, there was a certain amount of mistrust of high school GPA as an indicator of 
college readiness. As has been found in other settings (Hetts, 2016), interviewees had 
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concerns about whether high school GPA was as good a predictor as a score on a standardized 
test such as the ACCUPLACER. For example, one interviewee commented, “Also, just one 
other thing is — I’m wondering if the GPAs at the various schools can be really seen as being, 
quote, equal.” 


Changes to College Processes and Offerings 


The procedural workload at the college sometimes increased, especially at the outset 
of the project. In most cases, considerably less work was required to sustain the alternative 
placement system once it was established (also see Chapter 5, Cost Analysis). 


At colleges where the proportion of students in developmental and college-level 
courses changed considerably due to the new placement system, extra work was required to 
adjust classroom and faculty assignments. Department chairs reported that they had to make 
changes based on the different numbers of college developmental and college-level sections 
that were needed. In some cases, different classrooms had to be scheduled based on lower 
caps on the numbers of students allowed in developmental courses. At some colleges, there 
was a concern about whether all of the current faculty would be needed and about whether 
some courses, especially those in developmental reading, would continue to exist. 


Changes to Testing 


As noted previously, testing center staff and counselors noted that there were 
disadvantages to not having placement information immediately available to students under 
the alternative system. In some cases, the delay was viewed as a minor inconvenience. In 
other cases, there were concerns about making students come a second time for counseling 
or about missing an opportunity for testing center personnel to provide them with initial 
guidance on what placement results meant. 


These students were used to getting the result, and they want the 
results right away, and we have to tell them, “You have to wait until 
the next business day.” It was a lot more follow-up and phone calls 
to the office, instead of when you had them there. 


Placement Results 


There were concerns among some college staff about the impact on students of being 
placed too high or too low. The data analytics placement system placed some students into 
remedial courses even if they achieved what had been considered “college-level” scores on 
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the placement test. College staff found it difficult to tell students that they could not take 
college-level courses in these situations. '4 


The student scored above our cutoff for the math but was placed 
into a [remedial] math course. So I guess we were confused because 
I guess that we thought the purpose was to benefit the students. 


In other cases, college staff worried about students being placed into college-level 
courses where they would struggle and possibly fail. 


I just get so concerned for those 126 people who get jumped to 
College Composition. I’m like, “At least give them a workshop or 
something,” you know? 


In some colleges, access to programs or majors was affected by placement 
determinations. For example, some student support programs explicitly targeted students in 
developmental courses; their enrollments could be affected by fewer students placing into 
these courses. 


Like I said, [TRIO] programs; it’s going to destroy what we do. 


In other cases, selective majors at some colleges were open only to students who were 
deemed college-ready. Some college personnel were concerned that students placed into 
developmental coursework through the alternative system would not be eligible to enter them. 


Maybe that student is trying to get into one of these high-demand 
majors. Now that we’re saying that they have to take a remedial 
class, they’re no longer eligible for that. 


Lessons for Others 


We asked college representatives about what advice they would give to colleges that 
were considering the use of a multiple measures, data analytics placement system. Their 
responses are summarized here. 


Recognize that the goal of improved assessment and placement is worthwhile. 
Despite the difficulties that were encountered with implementation, most interviewees said 
that they would advise others to take on this work. Many believed that a placement system 
using multiple measures would be a more effective way to place students and that students 
would begin college in the courses that would benefit them the most. 


‘4Whenever possible, placement test scores were not shown to college staff to avoid this 
circumstance, but there were cases where both test scores and placement results were visible. 
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I think that we’re going to have a lot more students in college-level 
courses who wouldn’t have been before, and they’re going to be 
successful. 


In addition, one interviewee pointed out that the college was positively viewed in a 
recent accreditation process because of its participation in this project. 


Be sure to have sufficient time to engage with varied groups across campus to 
establish buy-in. At most of the colleges, there was a sense that the project should have been 
rolled out more slowly and that a wider range of stakeholders should have been engaged. 
Most of the interviewees believed that at least some constituencies had not had enough time 
to understand the data analytics placement system and the implications for the college. They 
advised new implementers to take the time to communicate fully with different groups. One 
suggested an internal marketing campaign to build understanding, buy-in, and excitement 
about the project. Others were more focused on making sure that everyone was adequately 
trained in their changed roles. 


Make sure you’re involving the right parties. Make sure the 
decision makers are sitting around the table, and make sure they 
understand the decisions they’re making. 


Anticipate possible problems that could arise, and craft solutions for them. A 
number of interviewees suggested that new implementers spend time coming up with varied 
scenarios to troubleshoot before a new placement system is launched. 


[We tried] to think of as many scenarios as possible. We realized 
going in that there probably would be something that would come 
up that we hadn’t thought of. That's just human nature. 


Make plans ahead of time for obtaining high school transcripts and entering the 
data. A suggestion was made that colleges do the necessary work with local high schools to 
obtain transcripts before starting a new placement system that incorporates this information. 
Likewise, the point was made that colleges need to plan for the data entry time required to 
record transcript information in a useful way. 


Calculate the likely impact on course schedules, and be prepared to make 
necessary changes. One interviewee suggested that colleges predict how their needs for 
course sections are likely to change when a new placement system is in place, taking into 
account faculty assignments and room sizes. 

If the model looks like it’s going to be placing more people at a 
higher level, that will have implications, and you may want to start 


thinking about whether you have to add more sections.... Who’s 
going to teach those sections? And get it ready.... 
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Reduce the burden on students as much as possible. Several personnel suggested 
planning ahead to make sure that services to students are not negatively affected. In particular, 
interviewees were concerned with minimizing the delay in providing placement information, 
especially during one-day student enrollment events. 


Think about sustainability from the beginning. Some interviewees suggested 
planning beyond the short term, both to ensure that the alternative placement system has time 
to work and to have resources available to sustain it if it proves successful. 


I guess my advice would just be, if you decide to go to the multiple 
measures, let it run its course for multiple semesters before you 
decide to switch away from it. 


I’ve been here for a long time and know that these things come and 
go. I’m always willing to try something new that’s going to work 
for the students, but we need to follow through on that and see if it 
was beneficial. Should it be continued? Can it be institutionalized? 
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Chapter 4 
Early Impacts Data, Analysis, and Results 


In this chapter, we discuss the analytic sample, the approach used to calculate early 
impacts, and the results obtained. The early impacts results for this report draw on first-term 
outcomes data from the first cohort of sample students, who attended five of the seven SUNY 
community colleges participating in this study. '° 


Data and Sample 


The data used to place students and track their outcomes in this study come from two 
main sources: placement records and administrative data from each college. Student-level 
placement records include indicators for students’ actual placement levels in math and 
English, as well as information that is needed to determine students’ placements, regardless 
of assignment to either the program group or the control group. Placement records from each 
college contain high school GPAs and scores on individual ACCUPLACER tests. Additional 
variables included in the placement records vary by college. Examples of additional variables 
incorporated for certain colleges include the number of years between high school completion 
and college enrollment, type of diploma (high school diploma vs. GED), SAT scores, and 
New York State Regents Exam scores. 


In addition to placement records, college administrative data were collected for any 
student in the study who enrolled in at least one course during the fall 2016 term. These data 
include demographic information, such as gender, race/ethnicity, age, and financial aid status; 
semesters enrolled; courses taken, including course levels; credits attempted and earned; and 
course grades. 


We present findings from the study’s first cohort of students, which includes all 
eligible students who went through intake at a participating college in the fall 2016 semester 
and opted to participate in the study. This sample excludes students who were still enrolled 
in high school at the time of intake, those who took their first placement test outside of the 
intake period for fall 2016, and those whose ACCUPLACER or writing scores on a college- 


The two colleges excluded from this analysis did not begin implementing the random assignment 
procedures until after the fall 2016 intake period. 
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created test placed them into an English as a second language (ESL) course.'® Our final 
analytic sample for this early impacts analysis consists of 4,729 students who took a 
placement test at the five colleges at the time of fall 2016 entry, of whom 3,865, or about 82 
percent, enrolled in at least one developmental or college-level course of any kind during the 
fall 2016 term. 


In spring 2017, CAPR worked with the colleges and with SUNY to transfer initial 
data on students who had not opted out of the study and who took placement tests. For all 
colleges and SUNY, a list of all students enrolled in the study was provided. Institutional 
research personnel then delivered placement data for all students and transcript and 
demographic data on students who actually enrolled in any course in the fall 2016 term. This 
method of data extraction also allowed us to account for students who tested but who did not 
subsequently enroll. 


Analytic Method 


To test the hypothesis that a multiple measures, data analytics placement system 
differs from a single test placement system, we conducted an intention-to-treat analysis by 
comparing the average outcomes for students assigned to the program and control groups. 
Specifically, we estimated the following ordinary least squares regression: 


(6) Y¥, =a+ BR, + AQ; + NX, + 6Z; + €j 


where Yj represents short-term academic outcomes for student i; Rj indicates whether the 
individual was randomly assigned to be placed using the predictive algorithm; qj; is an 
indicator for the institution a student attends; Xj is a vector of baseline covariates, including 
gender, race/ethnicity, age, and financial aid status; Z; includes both math and English 
algorithm calculations for each student (which are essentially two indices for academic 
preparedness); and g; is a random error term. The coefficient of interest is B,'’ the effect of 


‘For colleges in which assessment and placement are managed outside of the ACCUPLACER 
portal (Colleges 2 and 3), we also exclude students who took their placement test over several days and/or 
retested in any subject. Due to a data collection error, we did not have access to complete placement 
information for these 45 students and therefore dropped them from our analysis. The results presented in 
this report are robust to the inclusion of this subgroup of students in the analytic sample. 

Given random assignment, ordinary least squares estimations of B ought to provide unbiased 
estimates of intention-to-treat effects. However, differential attrition across program and control groups 
could bias these estimates. Appendix Table A.6 shows test results for differential attrition across program 
and control groups by estimating Equation 7 without controls on an indicator for whether or not a student 
enrolled in any course at the college in which he or she took a placement test. There is no evidence of 
differential attrition from the sample. 
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assignment to the alternative placement system on exploratory outcomes at the end of the 
first (proximal) semester (see Appendix Table A.1). 


Baseline Student Characteristics 


Appendix Table A.7 shows baseline descriptive statistics for enrolled students in the 
sample, those who enrolled in any course in the fall 2016 term at one of the five colleges 
included in the early impacts analysis.'* Pre-randomization characteristics for the fall 2016 
enrollment cohort are reported in the first column of the table, and additional columns present 
results for each of the colleges separately. Fifty-two percent of enrolled students in the sample 
were male, and 43 percent were White. About half of all sample students enrolling in at least 
one course during the fall 2016 term received a federal Pell Grant for that term. 


Appendix Table A.7 indicates that there is some variation in demographic 
characteristics across colleges. For instance, Colleges 1, 2, and 3 have a greater proportion of 
White enrolled sample students than do Colleges 4 and 5, which have a higher share of 
Hispanic enrolled sample students. Using Pell Grant receipt as a proxy for low-income status 
suggests that average family income for enrolled sample students also varies across colleges. 
While Pell Grant recipients comprise more than 60 percent of enrolled sample students from 
Colleges 1, 2, and 3, they represent less than half of the enrolled sample students from 
Colleges 4 and 5.'° 


Post-Randomization Student Characteristics 


As previously discussed in Chapter 2, CAPR researchers worked with the colleges to 
develop college-specific data analytics algorithms, specify cut points on algorithm scores to 
establish decision rules for placement, and implement the resulting alternative placement 
system. Incoming students were randomly assigned to be placed into math and English 
courses using either the status quo placement system (control group) or the alternative 
placement system (program group). The random assignment procedure should, in 
expectation, ensure that students assigned to the program group are similar in all ways to 
those assigned into courses under the status quo placement system. Appendix Table A.8 
provides evidence that participants’ demographic and academic characteristics, including 
indicators for missing characteristics, are indeed well balanced across program and control 
groups for the fall 2016 study cohort. Students’ individual ACCUPLACER test scores also 


'8Recall that demographic information was not obtained for students who completed a placement 
test but did not subsequently enroll in a course at a college during the fall 2016 term. 

'In part, the small proportion of Pell Grant recipients in Colleges 4 and 5 can be explained by the 
exclusion from the final analytic sample of students who placed into ESL courses in Colleges 4 and 5, as 
these students are significantly more likely to be Pell recipients than their peers. 
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are similar across both groups.”° Overall, the magnitude of differences between program and 
control groups is small and statistically insignificant, providing reassurance that randomized 
treatment assignment was implemented as intended. 


Program Placement: Descriptive Outcomes 


Because the multiple measures, data analytics placement system uses a different set 
of criteria than the status quo system, we might expect at least some changes in placement 
level in math and English?! among program group students.** Importantly, however, any new 
placement procedure will not change the placement of some students. Figure 4.1 shows the 
observed changes in placement for those students in the fall 2016 cohort who took a 
placement test in each subject area. (For additional details, see Appendix Table A.9.)* Of 
the 2,455 students assigned to the program group, 92 percent took a placement test in math, 
and 76 percent took a placement test in English. Among those students who took a math 
placement test, 21 percent experienced a math placement different from what would have 
been expected under the status quo placement rules; 14 percent were placed into a higher 
level math course (1.e., a college-level course) than would have been expected under the status 
quo system, and 7 percent were placed into a lower level math course (i.e., a developmental 
course).7* 


Of the 76 percent who took a placement test in English, 48 percent of program group 
students experienced a change in their level of English placement (1.e., moved between a 
developmental and college-level placement); 41.5 percent placed into a higher level course, 
and 6.5 percent placed into a lower level course than they would have under the status quo 
placement system. 


ln order to avoid post-treatment bias, we report on and use only the student’s first 
ACCUPLACER score received on each subject test. 

*1Students were placed into both reading and writing developmental or college-level courses using 
the algorithm developed for English. 

24 change in placement is defined as a student being placed into a developmental course rather a 
college-level course or vice versa. A change in the level of either developmental or college-level course 
placement is not considered a change in placement for purposes of this analysis. 

Unlike the intention-to-treat analysis discussed below, the descriptive calculations that yield 
these results do not control for any college or student characteristics. 

4Students who were exempt from placement by test for math or English were not included in 
analysis for that subject area. Taking a placement test is defined as taking one or more placement subtests 
in a subject area. 
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Figure 4.1 


Observed Difference in Placement Relative to Status Quo 
Among Program Group Students Who Took a Placement Test in Each Subject Area 
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In examining the data, we find that students generally entered the courses into which 
they were placed, if they enrolled in math or English courses at all. Compliance rates were 
high across math and English, ranging from 88 to 99 percent across participating colleges. 
Instances of noncompliance can be at least partially explained by the fact that we only 
consider initial placements — retesting may have changed final placements in some cases. 


Treatment Effects 


To test whether program assignment affected student outcomes, we constructed four 
estimation models that build upon each other: (1) We first calculated a simple regression, 
including only college fixed effects, to estimate the relationships between the treatment 
indicator and each outcome; (2) we then added controls for the full set of predefined 
demographic characteristics for gender, race/ethnicity, and age included in Appendix Table 
A.7; (3) we then added proxies for income, including Pell and TAP Grant”? recipient status; 
and (4) we finally added the calculated math and English algorithm values for all students. 


The Tuition Assistance Program (TAP) is a need-based grant available to New York State 
residents who enroll as a first-time student at an approved postsecondary institution in the state. 


35 


Appendix Tables A.10 through A.18 show the results for each outcome of interest, 
including college-level placement in math and English, enrollment in college-level math and 
English, enrollment in and completion (with a grade of C or higher) of college-level math 
and English, enrollment in any college-level course, enrollment in and completion (with a 
grade of C or higher) of any college-level course, and college-level credits earned.”° 
Importantly, because we present intention-to-treat results, students who did not enroll in any 
courses following placement are nonetheless included in the sample and were coded with a 
zero on all enrollment, completion, and credit accumulation outcomes. Because the impacts 
shown in the results are based on a sample of students that include those who never entered 
courses, they may understate the impacts on students who did in fact enroll. 


Results are robust to each model specification. For the remainder of this section, we 
discuss results from Model 4, the preferred specification. 


Math 


Figure 4.2 shows the treatment effects, or the differences in outcomes between 
program and control group students, on college-level math placement, enrollment, and 
completion rates (with a grade of C or higher) in college-level math in the first term, among 
the 4,371 students in the first cohort who took a math placement test. For all three outcomes, 
there was a positive and statistically significant (p < .01) effect of being assigned to the 
program group. Students in the program group were 5.0 percentage points more likely than 
those in the control group to be placed in a college-level math course, 4.7 percentage points 
more likely to enroll in a college-level math course, and 3.1 percentage points more likely to 
enroll in and complete a college-level math course during their first term, after controlling 
for the full set of covariates. 


These effects may also be stated in proportional terms by dividing the percentage 
point differences in outcomes by the control group outcomes. Stated in this way, the treatment 
increased the probability of college-level math course placement by 11.4 percent, the 
probability of college-level math course enrollment by 18.6 percent, and the probability of 
college-level math course enrollment and completion by 22.0 percent. All these impact 
findings are statistically significant, meaning that it is highly unlikely that they are due to 
chance. The difference between placement and enrollment into a college-level math course 


26 Analyses presented in the current report use data from the first cohort of students’ first semester 
and therefore can be considered an initial testing of the effects on outcomes of the alternative placement 
system. Academic outcomes for all three cohorts of students will be tracked for three to five semesters after 
random assignment, depending on when the cohort entered the study. Future analyses will evaluate each 
outcome at the second through fifth (distal) semesters. 
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can be partially explained by the fact that some students who placed into college-level math 
did not take a math course during their first term.”’ 


Figure 4.2 
Math Outcomes (Among Students Who Took a Math Placement Test) 
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English 


There are similarly positive, though substantially larger, statistically significant (p < 
.01) impacts for English placement, enrollment, and completion (with a grade of C or higher) 
among the 3,533 students in the first cohort who took an English placement test for fall 2016 
entry. As shown in Figure 4.3 below, students who were in the program group were 30.4 
percentage points more likely to be placed in a college-level English course, 19.3 percentage 
points more likely to enroll in a college-level English course, and 12.5 percentage points more 
likely to enroll in and complete a college-level English course in the first term. The treatment 
therefore increased the probability of college-level English course placement by 58.0 percent, 
the probability of college-level English course enrollment by 47.3 percent, and the probability 
of college-level English course enrollment and completion by 46.0 percent. As in the case for 


?7Among students who took a math placement test and placed into but did not enroll in a college- 
level math course, 31.5 percent did not enroll at any course at the college, and 65.1 percent did enroll in at 
least one course but did not enroll in a math course. 
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math, the difference between placement and enrollment into a college-level English course 
can be partially explained by the fact that some students who placed into college-level English 
did not take an English course in their first term.”* 


Figure 4.3 
English Outcomes (Among Students Who Took an English Placement Test) 
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Any College-Level Course 


In addition to subject-specific impacts, we also test whether program or control group 
status had any impact on overall college-level course taking. The entire sample is used in this 
analysis (4,729 students). As Figure 4.4 shows, after controlling for all covariates of interest, 
assignment to the program group increased the probability of enrolling in any college-level 
course by 0.9 percentage points (or 1.1 percent) and increased the probability of enrolling 
and completing (with a grade of C or higher) any college-level course by 4.2 percentage 
points (6.8 percent). Both effects are statistically significant (p < .01). Despite the relatively 
small impacts on overall college-level course enrollment, it is important to recognize that the 
alternative placement system may nonetheless have placed students into college-level math 


?8Among students who took an English placement test and placed into but did not enroll in a 
college-level English course, 58.2 percent did not enroll at any course at the college, and 37.5 percent did 
enroll in at least one course but did not enroll in an English course. 
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and English courses who were better able to succeed in them. The greater (6.8 percent) 
increase in overall college-level course completion than in enrollment (1.1 percent) is 
consistent with this possibility. 


Figure 4.4 
College-Level Course Outcomes (Among All Students) 
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Credit Accumulation 


Finally, Figure 4.5 shows the impact that assignment to the program group had on 
college-level credits accumulated in the first term. Students assigned to the program group 
earned, on average, 0.60 more college-level credits than students in the control group (p < 
.01). This increase represents an increase of 11.6 percent in the number of college-level 
credits earned over the control mean of 5.17 college-level credits earned, which suggests that 
the multiple measures, data analytics placement system had some impact on overall college- 
level credits earned in the first term. This finding compares favorably with evaluation 
findings from other initiatives designed to increase student success in college. For example, 
an experimental study of summer bridge programs found that students who had participated 
in such programs earned an additional 0.6 credits in college after two full years (Barnett et 
al., 2012). 
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Figure 4.5 
College-Level Credit Accumulation (Among All Students) 
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Subgroup Analyses 


To test whether program assignment led to differential treatment effects, we conduct 
subgroup analyses by race/ethnicity, Pell recipient status, and gender and test the significance 
of interaction effects between treatment status and each subgroup. The detailed results of the 
subgroup analyses are presented in Appendix Tables A.19 through A.36.”? Because we are 
limiting this analysis to only those students who enrolled in any course at the college 
(“enrolled students”) — a post-random assignment characteristic — these analyses are no 
longer causal and may produce biased estimates of treatment effects. This would be apparent 
if the treatment status differentially impacted enrollment patterns by subgroup. We test for 
this possibility (see Appendix Table A.8) by examining whether enrollment is balanced 
across treatment status and whether this is true for each subgroup. Post-randomization 
balance across subgroups conditional on college enrollment offers some assurance that there 
are not differential treatment impacts on college enrollment by subgroup. Nonetheless, the 
results described below should not be considered strictly causal. It is also worth noting that 
observed sample sizes may limit our power to detect statistically significant effects for 


2° Although not presented here, estimated interaction effects for each subgroup are available upon 
request. 
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smaller subgroups.*° The larger sample sizes that will be used in the final report analyses may 
reveal additional differential impacts by subgroup. 


Math 


We find that gains in college-level math placement, enrollment, and enrollment and 
completion (with a grade of C or higher) were experienced by students assigned to the 
program group for most of the subgroups we considered (p < .1); the exceptions are that we 
find no statistically significant treatment impacts for men across all math outcomes 
considered and also find no statistically significant impacts on course completion for Black 
and White students. Importantly, small sample sizes may explain null findings for some 
subsamples. For example, the minimum detectable effect (MDE) given the observed 
precision and sample size for Black students is 8 percentage points.*! 


With the exception of the gender subgroups, interactions between the treatment status 
and each of the subgroups we considered are generally small and not statistically significant. 
This suggests that gaps in placement, enrollment, and completion rates in math between 
subgroups (other than gender subgroups) may not have been affected by the treatment. Stated 
another way, the statistically insignificant interactions suggest that the treatment may not 
have differentially impacted students by race/ethnicity or Pell Grant status. 


However, unlike the other subgroups considered, women appear to have benefitted 
more from program group status than men across all math outcomes considered. Placement 
by the alternative system narrowed gender-based placement, enrollment, and enrollment and 
completion gaps in math at levels that are statistically significant. Use of the alternative 
placement system reduced the gap in college-level math placement between men and women 
by 7.7 percentage points (from a gap of 8.6 percentage points to a gap of 0.9 percentage 
points), reduced the gap in college-level math enrollment by 5.0 percentage points (from a 
gap of 12.2 percentage points to a gap of 7.2 percentage points), and reduced the gap in 
college-level math enrollment and completion by 4.5 percentage points (from a gap of 4.9 
percentage points to a gap of 0.4 percentage points). Statistically significant interaction 
effects provide evidence that placement by the alternative system narrowed gender-based 
placement (p <.01), enrollment (p <.1), and enrollment and completion gaps (p< .1) in math. 
Gender and other subgroup results in math are summarized in Figures 4.6 through 4.8 below. 


3°Appendix Figure A.1 shows the relationship between sample size and the minimum detectable 
effect (MDE) based on 80 percent power, 5 percent significance, and an 7’ equal to 0.1. 

3!MDE is calculated based on 80 percent power and a 5 percent significance level for a two-tailed 
test. 
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Figure 4.6 
Placement in College-Level Math (Among Enrolled Students) 
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Figure 4.7 
Enrollment in College-Level Math (Among Enrolled Students) 
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Figure 4.8 
Enrollment in and Completion of College-Level Math (Among Enrolled Students) 


100% 5 


0% 


Black Hispanic White Pell Non-Pell Female Male 


=Control group Program group 


***D < 01, **p < .05, *p <.10. 


English 


We also conduct subgroup analyses for each of the three college-level English 
outcomes discussed in the main analysis. Figures 4.9 through 4.11 below summarize the 
results of these analyses. Overall, we find positive and statistically significant impacts on 
college-level English placement, enrollment, and enrollment and completion (with a grade of 
C or higher) for each of the subgroups we consider (p < .01). Although significance testing 
on interaction effects in many cases fails to reveal differential impacts by subgroup, 
statistically significant interaction effects provide suggestive evidence of larger treatment 
effects on college-level English placement and enrollment for Black and Hispanic students 
compared with White students (p < .01). 


White students in the control group had higher outcomes than Black and Hispanic 
students in the control group, but under program group status, the racial/ethnic gaps in the 
placement and enrollment outcomes narrowed or even reversed. The Black-White gap in 
rates of placement in college-level English narrowed from 19.1 to 1.2 percentage points. (The 
White control group rate was 60.4 percent.) The equivalent Hispanic—White gap reversed 
from 1.2 percentage points in favor of White control group students to 6.6 percentage points 
in favor of Hispanic program group students. 
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The Black-White gap in rates of enrollment in college-level English reversed from 
15.4 percentage points in favor of White control group students to 1.1 percentage points in 
favor of Black program group students. (The White control group rate was 56.3 percent.) The 
equivalent Hispanic—White gap reversed from 4.8 percentage points in favor of White control 
group students to 7.0 percentage points in favor of Hispanic control group students. 


Models testing the interactions between treatment status and indicators for 
race/ethnicity thus suggest that placement by the alternative system may have narrowed or 
reversed college-level English placement and enrollment gaps in favor of racial/ethnic 
minorities. It is important to emphasize that we do not find evidence that the enrollment and 
completion gap narrowed between White and racial/ethnic minority groups. 


Figure 4.9 
Placement in College-Level English (Among Enrolled Students) 
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Figure 4.10 
Enrollment in College-Level English (Among Enrolled Students) 
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Figure 4.11 
Enrollment in and Completion of College-Level English (Among Enrolled Students) 


100% 5 


Pres a ss ek K 
52.1 2k 52.1 cee 
aa 49.8 45.4 50.9 46.7 


Black Hispanic White Pell Non-Pell Female Male 


=Control group Program group 


#8 < 01, **p <.05, *p <.10. 


45 


Chapter 5 
Cost Analysis 


In this chapter, we provide an estimate of the first-term costs of implementing the 
multiple measures, data analytics placement system at the five colleges that began to invite 
students into the study in fall 2016, as well as an estimate of the subsequent costs of operating 
the system after the first year. The cost estimates of the alternative placement system are 
relative to the cost of the status quo testing system for placement. 


First-Year Fall-Term Costs 


Relative to the status quo system, there are new resource requirements for: (1) 
performing administrative setup and collection of data for the data analytics algorithms; (2) 
creating the algorithms (one each for math and English) for implementation; and (3) running 
the alternative placement system at the time of placement testing. For both the status quo 
system and the new placement system there are costs in (4) administering placement tests — 
although the costs per test may differ between the status quo and the new system. Also, for 
both options, there may be future resources required when students (5) progress into college- 
level courses after completing developmental coursework. If more students progress into 
college-level courses, colleges will have to provide extra courses. Currently, no information 
on differences in college-level coursework across the two options is available, so the costs of 
this last resource category cannot be calculated. 


The costs are calculated for the five colleges using the ingredients method (Levin et 
al., 2017). Costs are derived from the inputs used at each college, multiplied by standardized 
prices per input. All costs are expressed in 2016 dollars. Information on ingredients was 
collected from direct interviews with personnel who implemented the new testing protocols; 
at three colleges, two additional sets of interviews were undertaken. Information on input 
prices and overhead costs were collected from secondary sources. The resulting cost estimates 
are the expected cost of implementing the new placement system at a college of similar size 
and organization as the five sample colleges. 


The cost per college is given in Table 5.1. (See table notes for details.) Resources for 
personnel to implement the alternative placement system are divided into several groups. 
Within the college, there are IT/computing staff, responsible for creating new computing and 
data infrastructure (and for some data management); program staff, responsible for cleaning 
data and implementing the new placement system during and after the students take the 
placement tests, as well as for advising students about courses; senior staff and faculty, 
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responsible for introducing and managing the new system; and administrative staff, 
responsible for supporting the implementation of the new system. 


Outside the college, there is personnel time from the research team that developed the 
data analytics algorithms and (at some sites) applied the algorithms to determine student 
placement into developmental or college-level courses. These external costs are allocated 
evenly across the five sites. (Also, there are personnel at the SUNY system-level office who 
committed time to the project. Currently, these costs are not available and so are not included 
in estimates in Table 5.1.) In addition to personnel costs, there are also fringe benefits and all 
other operating costs (overhead, facilities, and materials costs). Finally, there are costs for 
administering the placement tests; based on their direct analysis, Rodriguez, Bowden, 
Belfield, and Scott-Clayton (2014) estimated these costs at $30—44 per test. 


Across the five colleges, the total cost to fully implement the new system was 
$762,640 across the 5,303 students* in a single (fall 2016) cohort. However, this amount 
includes the cost of administering the placement tests, a cost that is also incurred under the 
status quo placement system. Therefore, the net cost of the new system is $603,550 per 
cohort, or $110 per student. 


As shown in Table 5.1, the cost per student varies between $70 and $320. We assume 
that each student enters the college once, so these per-unit costs apply to each eligible student 
on initial intake. The variation in per-student cost is primarily driven by the number of 
students at each site. (Note the markedly lower numbers of students at high-cost Colleges 2 
and 3 in Table 5.1.) More enrollments lead to lower costs per student because the costs of 
creating the algorithms for the new system are mostly fixed — that is, they do not vary with 
the number of students involved. Also, costs per college varied depending on how much 
information was previously available to determine the new placement algorithms and how 
many students had the requisite information. Data entry costs were lower if the college had 
all high school information preloaded into its databases; in contrast, data entry costs were 
higher if each student’s information had to be entered into the computer system individually. 


Interviewees did not indicate significant resource changes with respect to instruction. 
Potentially, the new placement system may change assignments such that more students are 
now in college-level courses; this would require more college-level faculty and perhaps more 
sections of college-level courses. However, most colleges indicated that faculty could be 
reassigned from teaching developmental courses to teaching college-level courses; also, few 
changes in class size were anticipated. 


32Some of these students, including dual enrollment and ESL students, were excluded from the 
analytic sample of 4,729 students. 
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Subsequent-Year Fall-Term Costs 


The data analytics placement system requires an initial investment to create the 
algorithms that are used to assess students. This investment is not necessary each semester. 
For ongoing operations, modest resources are required to administer the new placement 
system. First, resources are needed for collection of information from entering students for 
use in the algorithms. Second, limited additional personnel time is needed to assign students 
to developmental education or directly to college-level courses. 


Using the same number of students as in the first semester, we estimate the costs of 
operating the system in the next fall term, after the algorithms have been developed. The 
estimates are inexact because we do not yet know what resources will be used. As shown in 
Table 5.2, the operating cost per student per semester over the status quo is estimated at $40. 
This amount is less than half of the total cost to implement the data analytics placement 
system in the first semester. However, this estimate is not precisely bounded: It may be as 
low as $10 or as high as $170. 


Future Analysis on Cost-Effectiveness 


The cost per student reported above is the cost of employing the data analytics system 
relative to the status quo system. This is called the direct cost. When information on the 
outcomes of the alternative system is available, cost estimates can be used as part of a cost- 
effectiveness analysis. However, for cost-effectiveness analysis, both direct and indirect costs 
should be included. In this case, the indirect costs are the costs of progression into college 
courses (Resource Category 5). These indirect costs will be calculable once there is 
information on how program and control group students progress through college. This 
analysis will be included in the final report. 
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Table 5.1 


First-Year Fall-Term Implementation Costs for the Data Analytics Placement System 


Ingredient 

Item Price per FTE College 1 College 2 College 3 College 4 College 5 Total 
Personnel (FTEs) 

IT? $56,230 0.18 0.10 0.18 0.05 0.30 0.80 

Program? $47,500 0.44 0.91 0.15 0.14 0.87 2.50 

Senior/faculty° $62,500 0.16 0.07 0.25 0.25 0.25 0.98 

Administrative support* $35,950 0.05 0.05 0.05 0.05 0.05 0.25 

Evaluator time® $56,230 0.10 0.10 0.10 0.10 0.10 0.50 
Total personnel costs $48,600 $60,160 $40,180 $32,470 $81,000 $262,410 
Fringe benefits‘ $16,040 $19,850 $13,260 $10,720 $26,730 $86,600 
Overheads/facilities® $47,140 $58,360 $38,970 $31,500 $78,570 $254,540 
Placement test administration® $10,770 $12,930 $21,540 $31,260 $82,590 $159,090 
Total cost $122,550 $151,300 $113,950 $105,950 $268,890 $762,640 
Total cost over status quo $111,780 $138,370 $92,410 $74,690 $186,300 $603,550 
Students per semester 359 431 718 1,042 2,753 5,303 
Average cost over status quo $310 $320 $130 $70 $70 $110 


SOURCE: Ingredients information on FTEs from interviews with key personnel at five colleges. 


NOTES: 2016 dollars. Present values (d = 3%). Rounded to $10. 

Salary data from https://www.cs.ny.gov/businesssuite/Compensation/Salary-Schedules/index.cfm?nu=PST &effdt=04/0 1/20 15&archive=1 &fullScreen 
bAnnual salary (Step 4, Grade 13) from https://www.suny.edu/media/suny/content-assets/documents/hr/UUP_2011-2017_ProfessionalSalarySchedule.pdf 
°Midpoint MP-IV from https://www.suny.edu/hr/compensation/salary/mc-salary-schedule/ 
¢https://www.cs.ny.gov/businesssuite/Compensation/Salary-Schedules/index.cfm?nu=CS A &effdt=04/01/2015&archive=1&fullScreen 

‘Estimated from timesheets by CAPR researchers. 

‘Uprated from ratio of fringe benefits to total salaries (Integrated Postsecondary Education Data System [IPEDS] data, 2013, 846 public community 
colleges). 

’Uprated from ratio of all other expenses to total salaries (IPEDS data, 2013, 846 public community colleges). 

Cost to administer placement test from Rodriguez et al. (2014). 
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Table 5.2 
Subsequent-Year Fall-Term Operating Costs for the Data Analytics Placement System 


Ingredient 

Item Price per FTE College 1 College 2 College 3 College 4 College 5 Total 
Personnel (FTEs) 

IT? $56,230 0.02 0.01 0.02 0.01 0.03 0.08 

Program? $47,500 0.44 0.91 0.15 0.14 0.87 2.50 

Senior/faculty* $62,500 0.02 0.01 0.03 0.03 0.03 0.10 

Administration! $35,950 0.01 0.01 0.01 0.01 0.01 0.03 
Total personnel costs $23,150 $44,140 $10,020 $8,640 $44,520 $130,470 
Fringe benefits® $7,640 $14,570 $3,310 $2,850 $14,690 $43,060 
Overheads/facilities $7,410 $14,130 $3,210 $2,760 $14,250 $41,770 
Placement test administration® $10,770 $12,930 $21,540 $31,260 $82,590 $159,090 
Total operating cost (TOC) $48,970 $85,770 $38,080 $45,510 $156,050 $374,390 
TOC over status quo $38,200 $72,840 $16,540 $14,250 $73,460 $215,300 
Students per semester 359 431 718 1,042 2,753 5,303 
Average operating cost over $110 $170 $20 $10 $30 $40 

status quo 


SOURCE: Ingredients information on FTEs from interviews with key personnel at five colleges. 


NOTES: 2016 dollars. Present values (d = 3%). Rounded to $10. 

“Salary data from https://www.cs.ny.gov/businesssuite/Compensation/Salary-Schedules/index.cfm?nu=PST &effdt=04/0 1/2015 &archive=1 &fullScreen 
>Annual salary (step 4, grade 13) from https://www.suny.edu/media/suny/content-assets/documents/hr/UUP_2011-2017_ProfessionalSalarySchedule.pdf 
°Midpoint MP-IV from https://www.suny.edu/hr/compensation/salary/mc-salary-schedule/ 
dhttps://www.cs.ny.gov/businesssuite/Compensation/Salary-Schedules/index.cfm?nu=CS A &effdt=04/01/2015&archive=1 &fullScreen 

‘Uprated from ratio of fringe benefits to total salaries (IPEDS data, 2013, 846 public community colleges). 

fUprated from ratio of all other expenses to total salaries (IPEDS data, 2013, 846 public community colleges). 

8Cost to administer placement test from Rodriguez et al. (2014). 

Operating cost refers to running of new placement system after initial algorithm has been developed and tested. 
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Chapter 6 
Conclusion 


This report is the first of two that will emerge from this random assignment study of 
a multiple measures, data analytics placement system used at seven SUNY community 
colleges for students who entered in the fall 2016, spring 2017, and fall 2017 terms. In this 
report, we describe how this alternative placement system — which uses high school 
transcript data and other information, in addition to placement test scores — was designed 
and developed, how it was implemented, how the use of the system affected the college 
community, and its cost. We also report on the first-semester impacts of the alternative 
placement system on the first cohort of 4,729 students who took a placement test at five of 
the seven colleges for fall 2016 entry. In the final report, to be published in 2019, we will 
report on the impact of the placement system on the full analytic sample of students — about 
13,000 persons who entered the seven colleges in all three cohorts. 


Most of our implementation findings are based on interviews with college personnel 
at all seven colleges. We found that the design and implementation of the alternative 
placement system was considerably more complex than initially expected by both the 
research team and participating staff at the colleges. Colleges often set up formal or informal 
committees to manage the work of developing and setting up the system; most also had a 
designated project lead. Steps needed to create the alternative system and get it to run 
smoothly took substantial time to accomplish. These included understanding how the new 
system would work; establishing buy-in from multiple stakeholders; developing the data 
analytics algorithms; deciding on cut points for placement; and creating procedures for newly 
required tasks such as data entry of high school transcript data, building and testing the 
technical infrastructure for alternative placement, and adjusting course offerings to match 
changes in placement. 


While these activities were demanding, every college was successful in overcoming 
barriers and developing the procedures needed to support the operation of the data analytics 
placement system for its students. Five colleges achieved this benchmark in time for 
placement of students entering in the fall of 2016, while the other two colleges did so in time 
for new student intake in the fall of 2017. Our results concerning implementation suggest that 
establishing a data analytics placement system is challenging but manageable. They also 
suggest that some colleges that want to implement a similar system may need technical 
assistance both to conduct analyses using historical student data that are necessary to create 
data analytics algorithms for math and English and to integrate these algorithms and other 
required placement components into the existing college placement computer infrastructure. 
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In our cost analysis, we estimate that implementing the alternative placement system 
added between $74,690 and $186,300 per college in first-year fall-term costs (an average of 
$110 per student) to the status quo costs for testing and placing students. Ongoing 
(subsequent-year fall-term) costs were much lower, ranging from $14,250 to $73,460 per 
college (an average of $40 per student) over status quo costs. In the final report we will have 
sufficient data to provide a cost-effectiveness analysis. 


Our early impacts findings are restricted to first-term impacts among the first cohort 
of students. While only exploratory, they suggest that the multiple measures, data analytics 
placement system used at the participating institutions holds considerable promise. Across 
the five colleges included in the analysis, a greater proportion of students in the first cohort 
who were placed using the data analytics system (those in the program group) were assigned 
to college-level courses in their first term than of those placed using the status quo placement 
system (those in the control group). In addition, a greater proportion of program group 
students enrolled in and passed (with a grade of C or higher) college-level courses in math 
and English. 


Our first-term subgroup analyses are not strictly causal and involve smaller sample 
sizes than our full sample analyses. Initial findings suggest that women benefitted more than 
men from program group status in math on all outcomes considered, and that Black and 
Hispanic students benefitted more than White students from program group status in English 
for placement and enrollment but not for completion (with a grade of C or higher). 


Key findings are summarized as follows: 


e Twenty-one percent of program group math students and 48 percent of 
program group English students were placed differently than they would 
have been under the status quo system. 


e Most of those program group students who were placed differently were 
placed higher than they would have been under the status quo system 
(i.e., were placed in college-level rather than developmental 
coursework). Among math students, 14 percent placed higher, while 7 
percent placed lower. Among English students, 41.5 percent placed 
higher, while 6.5 percent placed lower. 


e Program group students were 3.1 and 12.5 percentage points more likely 
than control group students to enroll in and complete (with a grade of C 
or higher) a college-level math or English course in the first semester. 
(The enrollment and completion rates among the control group were 14.1 
percent in math and 27.2 percent in English.) 
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e Program group students earned 0.6 more college credits in the first 
semester than did students in the control group (5.77 vs. 5.17 credits). 


e While men had higher math outcomes than women in both the control 
and program groups, women benefitted more from program group status 
in math on all three outcomes considered. For example, the male-female 
gap in the rate of enrollment in and completion (with a grade of C or 
higher) of college-level math narrowed from 4.5 percentage points 
among control group students to 0.4 percentage points among program 
group students. (The male control group rate was 19.5 percent.) 


e White students in the control group had higher English outcomes than 
Black and Hispanic students in the control group, but under program 
group status, the racial/ethnic gaps in both the rate of placement and the 
rate of enrollment in college-level English narrowed or even reversed. 
Yet we do not find evidence that program group status narrowed the gap 
in the rate of completion (with a grade of C or higher) of college-level 
English between White and Black or White and Hispanic students. 


These early results are broadly positive but assess outcomes based on merely one 
semester of data. Further impact analyses using additional data will be performed to evaluate 
the effects of using a multiple measures, data analytics system to place incoming students 
rather than a system based on scores on standardized placement tests alone. The final report 
on this study will examine a range of outcomes three to five semesters after students’ initial 
entry into college at seven SUNY community colleges. Our final sample of over 13,000 
students in a randomized controlled trial covering this time period will allow us to present 
impact findings in which we can have greater confidence than those provided here. Most 
important, we will be able to estimate over a greater time period student progress on our two 
most central (confirmatory) outcomes: completion of introductory college-level math and 
English courses and accumulation of college credits. We will also gain a better understanding 
of whether results differ by college and by student subgroup. 
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Appendix Table A.1 


Student Academic Outcome and Process Measures Used in the Evaluation 


Outcome/Process Measure 


Initial placement level changed 


Initial placement level 


Subject area courses attempted 


Subject area courses completed 


Subject area sequence completed 


Credits attempted 


Credits earned 


Completion 


Persistence 


Description of Measure 


Binary indicator that placement 
changed 


Categorical indicator of placement 
level 


Binary indicators of courses attempted 
(e.g., college-level, dev ed. 1, dev ed. 
2, etc.) 


Binary indicators of courses 
completed (e.g., college-level, dev ed. 


1, dev ed. 2, etc.) 


Completed introductory college- 
level (gatekeeper) course 


Number of college-level credits 
attempted 


Number of college-level credits 
earned 


Earned a degree or certificate 


Enrollment status in each subsequent 
semester 


Proximal 
(Semester 1) 


Exploratory 


Exploratory 


Exploratory 


Exploratory 


Exploratory 


Exploratory 


Exploratory 


Exploratory 


Exploratory 


Distal 
(Semesters 2—5) 


Exploratory 


Exploratory 


Exploratory 


Exploratory 


Confirmatory 


Exploratory 


Confirmatory 


Exploratory 


Exploratory 
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Appendix Table A.2 


College Characteristics’ 


Institution 
Characteristic Cayuga Jefferson Niagara Onondaga Rockland Schenectady Westchester. 
General college information 
Student population 7,001 5,513 7,712 23,984 10,098 8,458 22,093 
Full-time faculty 69 80 151 194 122 79 215 
Part-time faculty 170 177 0 480 409 0 2 
Student/faculty ratio 20 18 16 23 23 23 16 
% receiving financial aid 92 91 92 92 56 92 70 
Demographics 
Race/ethnicity (%) 
st Indian or Alaska 0 1 1 1 0 1 1 
Asian 1 2 1 3 5 7 4 
Black or African American 5 7 11 12 18 14 21 
Hispanic/Latino 3 11 3 5 20 6 32 
Native Hawaiian or other 0 0 0 0 0 1 0 
White 85 73 80 49 39 67 33 
ea aa race/ > 3 > 3 2 2 2 
Race/ethnicity unknown 3 3 1 27 15 2 5 
Nonresident alien 1 1 0 0 1 0 1 
Gender (%) 
Female 60 58 59 52 54 53 53 
Male 40 42 41 48 46 47 47 
Age (%) 
Under 18 30 17 19 24 10 37 1 
18-24 44 52 60 55 63 40 69 
25-65 26 31 21 21 26 23 30 
Unknown 0 0 0 0 0 0 0 
Retention/graduation rates (%) 
Full-time students 56 55 63 57 68 56 64 
Part-time students 28 30 47 34 56 50 53 
Three-year graduation rate 24 27 28 20 29 20 15 


Transfer-out rate 18 19 18 22 19 22 18 
@Based on fall 2015 IPEDS data. 
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Appendix Table A.3 


Math Algorithm Components by College 


Years Since HS Regents SAT ACCUPLACER ACCUPLACER 

High School High School Diploma/ Math Math Arithmetic ACCUPLACER _ College-Level 
College GPA Graduation GED Status Score Score Score Algebra Score Math 
College 1 xX > xX > xX xX 
College 2 > xX > > > xX xX xX 
College 3 x x x x x 
College 4 xX xX xX xX xX 
College 5 x x x x 

Appendix Table A.4 
English Algorithm Components by College 
Years Since ACCUPLACER WritePlacer or 
High School High School HS Diploma/ ACCUPLACER Sentence Skills Other Writing 

College GPA Graduation GED Status Reading Score Score Score* 
College 1 x x x x x 
College 2 xX > xX xX xX xX 
College 3 xX xX > xX xX 
College 4 Xx x Xx x x 
College 5 Xx Xx x x 


To test writing skills, some colleges administered WritePlacer, an ACCUPLACER subtest, while others administered a test 
created by the college. 
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Appendix Table A.5 


Historical Underplacement, Overplacement, and Total Error Rates* 


Subject Area Error Rates (%) College 1 College 2 College 3 College 4 
Overplaced 24.0 5.7 12.3 11.2 
Math Underplaced 8.3 44.7 29.1 36.0 
Total error rate 32.3 50.4 41.3 47.1 
Overplaced 12.0 15.2 13.7 8.4 
English Underplaced 30.7 29.8 33.7 43.7 
Total error rate 42.7 45.0 47.5 52.1 
Computed using student data from 2011-2014. 
Appendix Table A.6 


Effect of Program Assignment on College Enrollment 


College 5 
15.8 
18.5 
34.3 


10.7 
40.4 
51.1 


Treatment Status and Model Indicators (1) 
Program assignment -0.008 
(0.011) 
Control mean 0.821 
College fixed effects YES 
Demographic indicators NO 
Income indicators NO 
College preparedness measures NO 
Observations 4,729 


NOTES: Robust standard error in parentheses. 
***n < 01, **p <.05, *p <.10. 
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Appendix Table A.7 


Baseline Descriptive Student Characteristics by College (Among Enrolled Students) 


Overall College 1 College 2 College 3 College 4 College 5 
Std. Std. Std. Std. Std. Std. 
Characteristic Mean Dev. Mean Dev. Mean Dev. Mean Dev. Mean Dev. Mean Dev. 
Female (%) 48 50 58 49 48 50 52 50 50 50 44 50 
Race/ethnicity (%) 
White 43 49 83 38 78 42 74 44 38 49 31 46 
American Indian/Native Alaskan 1 8 1 10 2 13 4 19 0 6 0 if 
Asian 3 18 0 6 2 14 0 0 6 25 3 17 
Black 18 39 7 26 17 38 21 4] 22 42 18 38 
Hispanic 22 4] 7 25 0 0 1 9 30 46 25 43 
Pacific Islander 0 2 0 0 1 8 0 0 0 0 0 0 
More than one race/ethnicity 11 32 1 12 0 0 0 0 3 17 19 39 
Nonresident alien 0 3 1 9 0 0 0 0 0 3 0 0 
Race/ethnicity unknown 2 15 0 0 0 0 0 0 0 0 4 20 
Race/ethnicity missing 12 33 0 0 12 33 76 43 5 22 0 0 
Age at entry 19.94 5.57 20.10 6.77 22.06 7.70 21.03 6.15 20.43 5.89 18.99 4.25 
Age at entry missing (%) 0 0 0 0 0 0 0 0 0 0 0 0 
Pell Grant recipient (%) 49 50 64 48 66 47 61 49 36 48 47 50 
Pell Grant status missing (%) 5 22 0 0 0 0 37 48 0 0 0 0 
Total 3,865 327 408 673 1,002 2,319 
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Appendix Table A.8 


Post-Randomization Characteristics by Treatment Assignment 


Characteristic Mee Program Mean _ Treatment-Diff p-value Observations 
Female 47.50% 47.90% 0.40% 0.76 3,865 
Gender missing 17.80% 18.70% 0.90% 0.43 4,729 
Race/ethnicity 
White 43.50% 41.70% -1.80% 0.30 3,382 
American Indian/Native Alaskan 0.70% 0.70% 0.00% 0.81 3,382 
Asian 3.10% 3.70% 0.60% 0.34 3,382 
Black 16.90% 19.20% 2.30% 0.09 3,382 
Hispanic 21.90% 21.50% -0.40% 0.77 3,382 
Pacific Islander 0.10% 0.10% 0.00% 0.96 3,382 
More than one race/ethnicity 11.50% 10.90% -0.60% 0.57 3,382 
Nonresident alien 0.10% 0.10% 0.00% 0.61 3,382 
Race/ethnicity unknown 2.30% 2.10% -0.20% 0.76 3,382 
Race/ethnicity missing 28.30% 28.70% 0.40% 0.76 4,729 
Age at entry 19.9 20 0.10 0.52 3,593 
Age at entry missing 17.80% 18.70% 0.90% 0.43 4,729 
Pell Grant recipient 50.10% 53.30% 3.20% 0.06 3,672 
Missing Pell Grant info 4.40% 3.70% -0.70% 0.23 4,729 
TAP Grant recipient 39.10% 39.60% 0.50% 0.78 3,721 
Missing TAP Grant info 3.40% 3.40% 0.00% 0.99 4,729 
GED recipient 4.50% 4.20% -0.30% 0.57 4,600 
Missing GED status 2.90% 2.70% -0.20% 0.79 4,729 
High school GPA (100 scale) 78 78 0.00% 0.97 1,862 
High school GPA (missing) 59.80% 61.70% 1.90% 0.18 4,729 
ACCUPLACER subtest scores 
Arithmetic 45 45.9 0.90 0.26 3,439 
Algebra 53.1 53.7 0.60 0.77 4,407 
College-level math 35.5 35.4 -0.10 0.89 455 
Reading 72.3 71.9 -0.40 0.47 3,696 
Sentence skills 76.3 76.1 -0.20 0.12 1,072 
Writing 6.1 6.1 0.00 0.11 3,324 


Total 2,274 2,455 4,729 
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Appendix Table A.9 


Differences in Placement Relative to Status Quo for Program Group Students 


(1) (2) (3) (4) (5) (6) 
Exempt 
From 
Placement Took 

Test in Placement Same Placement Higher Lower 
Sample and Measure Subject Area Test Placement Changed Placement Placement 
Math Placement 
N 190 2,265 1,795 470 310 160 
Peeper wal 7.74 92.26 73.12 19.14 12.63 6.52 
program group sample 
Fo eoborsiuceuts : 100.00 79.25 20.75 13.69 7.06 
placed in math 
English Placement 
N 591 1,864 967 897 774 123 
Perens Onl 24.07 75.93 39.39 36.54 31.53 5.01 
program group sample 
Betee oy Or eens : 100 51.88 48.12 41.52 6.60 


placed in English 


65 


Appendix Table A.10 


Effect of Program Assignment on Placement in College-Level Math 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.044*** 0.047*** 0.049*** 0.050*** 
(0.014) (0.014) (0.014) (0.014) 
Control mean 0.437 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 4,371 4,371 4,371 4,371 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

***n < 01, **p <.05, *p <.10. 


Appendix Table A.11 


Effect of Program Assignment on Enrollment in College-Level Math 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.040*** 0.045*** 0.047*** 0.047*** 
(0.013) (0.012) (0.012) (0.012) 
Control mean 0.253 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 4,371 4,371 4,371 4,371 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

***¥n < 01, **p <.05, *p <.10. 
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Appendix Table A.12 


Effect of Program Assignment on Enrollment in and Completion of College-Level Math 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.028** 0.030*** 0.031 *** 0.03 1*** 
(0.011) (0.010) (0.010) (0.010) 
Control mean 0.141 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 4,371 4,371 4,371 4,371 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***¥y < 01, **p <.05, *p <.10. 


Appendix Table A.13 


Effect of Program Assignment on Placement in College-Level English 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.298 *** 0.301 *** 0.302*** 0.304*** 
(0.015) (0.015) (0.014) (0.014) 
Control mean 0.524 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 3,533 3,533 3,533 3,533 


NOTES: Robust standard errors shown in parentheses. The model from Column | includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

***¥n < 01, **p <.05, *p <.10. 
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Appendix Table A.14 


Effect of Program Assignment on Enrollment in College-Level English 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.183 *** 0.192*** 0.193 *** 0.193 *** 
(0.016) (0.014) (0.014) (0.014) 
Control mean 0.408 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 3,533 3,533 3,533 3,533 


NOTES: Robust standard errors shown in parentheses. The model from Column | includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

***n < 01, **p <.05, *p <.10. 


Appendix Table A.15 


Effect of Program Assignment on Enrollment in and Completion of College-Level English 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.118*** 0.124*** 0.125*** 0.125*** 
(0.016) (0.014) (0.014) (0.014) 
Control mean 0.272 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 3,533 3,533 3,533 3,533 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***¥y < 01, **p <.05, *p <.10. 
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Appendix Table A.16 


Effect of Program Assignment on Enrollment in Any College-Level Course 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.001 0.009*** 0.009*** 0.009*** 
(0.011) (0.003) (0.003) (0.003) 
Control mean 0.807 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 4,729 4,729 4,729 4,729 


NOTES: Robust standard errors shown in parentheses. The model from Column | includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

***n < 01, **p <.05, *p <.10. 


Appendix Table A.17 


Effect of Program Assignment on Enrollment in and Completion of 
Any College-Level Course 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.034** 0.041 *** 0.042*** 0.042*** 
(0.014) (0.011) (0.011) (0.011) 
Control mean 0.616 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 4,729 4,729 4,729 4,729 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

**D <.01, **p <.05, *p< .10. 
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Appendix Table A.18 


Effect of Program Assignment on College-Level Credits Earned 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Program assignment 0.503 *** 0.572*** 0:593*** 0.599*** 
(0.150) (0.130) (0.129) (0.129) 
Control mean 5.170 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 
Observations 4,729 4,729 4,729 4,729 


NOTES: Robust standard errors shown in parentheses. The model from Column | includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

**ED <O1, **p < 05, *p<.10. 
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Appendix Table A.19 


Effect of Program Assignment on Placement in College-Level Math by Race/Ethnicity 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Black students only 
Program assignment 0.078** 0.079** 0.085** 0.076** 
(0.039) (0.038) (0.038) (0.032) 
Control mean 0.357 
Observations 571 571 571 571 


Hispanic students only 


Program assignment 0.082** 0.092** 0.092** 0.097*** 
(0.037) (0.037) (0.037) (0.028) 

Control mean 0.481 

Observations 696 696 696 696 


White students only 


Program assignment 0.093 *** 0.094*** 0.095*** 0.096*** 
(0.025) (0.025) (0.025) (0.025) 

Control mean 0.493 

Observations 1,301 1,301 1,301 1,301 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

**n < 01, **p <.05, *p <.10. 
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Appendix Table A.20 


Effect of Program Assignment on Placement in College-Level Math 
by Pell Recipient Status 


Treatment Status and Model Indicators (1) (2) (3) (4) 


Pell recipients only 


Program assignment 0.067*** 0.070*** 0.071*** 0.071 *** 
(0.022) (0.022) (0.022) (0.022) 
Control mean 0.385 
Observations 1,763 1,763 1,763 1,763 
Non-Pell recipients only 
Program assignment 0.050** 0.048** 0.048** 0.044** 
(0.023) (0.023) (0.023) (0.018) 
Control mean 0.540 
Observations 1,633 1,633 1,633 1,633 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

***y < 01, **p <.05, *p <.10. 
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Appendix Table A.21 


Effect of Program Assignment on Placement in College-Level Math by Gender 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Female students only 
Program assignment 0.101 *** 0.103 *** 0.105*** 0.097*** 
(0.023) (0.023) (0.023) (0.019) 
Control mean 0.414 
Observations 1,701 1,701 1,701 1,262 
Male students only 
Program assignment 0.016 0.016 0.020 0.020 
(0.022) (0.022) (0.022) (0.021) 
Control mean 0.500 
Observations 1,862 1,862 1,862 1,862 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

**D < 01, **p < .05, *p< .10. 
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Appendix Table A.22 


Effect of Program Assignment on Enrollment in College-Level Math by Race/Ethnicity 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Black students only 
Program assignment 0.069* 0.070* 0.076** 0.069** 
(0.037) (0.036) (0.036) (0.032) 
Control mean 0.271 
Observations 571 571 571 571 


Hispanic students only 


Program assignment 0.080** 0.091** 0.092*** 0.096*** 
(0.036) (0.035) (0.035) (0.031) 

Control mean 0.328 

Observations 696 696 696 696 


White students only 


Program assignment 0.074*** 0.076*** 0.077*** 0.077*** 
(0.026) (0.026) (0.026) (0.026) 

Control mean 0.342 

Observations 1,301 1,301 1,301 1,301 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

**n < 01, **p <.05, *p <.10. 
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Appendix Table A.23 


Effect of Program Assignment on Enrollment in College-Level Math 
by Pell Recipient Status 


Treatment Status and Model Indicators (1) (2) (3) (4) 


Pell recipients only 


Program assignment 0.056*** 0.058*** 0.057*** 0.058*** 
(0.021) (0.021) (0.020) (0.020) 
Control mean 0.249 
Observations 1,763 1,763 1,763 1,763 
Non-Pell recipients only 
Program assignment 0.066*** 0.067*** 0.067*** 0.064*** 
(0.024) (0.023) (0.024) (0.021) 
Control mean 0.379 
Observations 1,633 1,633 1,633 1,633 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

***y < 01, **p <.05, *p <.10. 
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Appendix Table A.24 


Effect of Program Assignment on Enrollment in College-Level Math by Gender 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Female students only 
Program assignment 0.085*** 0.083 *** 0.086*** 0.081 *** 
(0.021) (0.021) (0.021) (0.020) 
Control mean 0.244 
Observations 1,701 1,701 1,701 1,701 
Male students only 
Program assignment 0.025 0.029 0.031 0.031 
(0.022) (0.022) (0.022) (0.022) 
Control mean 0.366 
Observations 1,862 1,862 1,862 1,862 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 


demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 


the previous controls plus calculated math and English algorithm values, which serve as proxies for college 


preparedness. 
***y < 01, **p <.05, *p <.10. 
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Appendix Table A.25 


Effect of Program Assignment on Enrollment in and Completion of 
College-Level Math by Race/Ethnicity 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Black students only 
Program assignment 0.039 0.039 0.042 0.037 
(0.030) (0.030) (0.030) (0.028) 
Control mean 0.145 
Observations 571 571 571 571 


Hispanic students only 


Program assignment 0.047 0.053* 0.051* 0.054* 
(0.031) (0.030) (0.030) (0.028) 

Control mean 0.184 

Observations 696 696 696 696 


White students only 


Program assignment 0.037 0.037 0.037 0.038 
(0.023) (0.023) (0.023) (0.023) 

Control mean 0.211 

Observations 1,301 1,301 1,301 1,301 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***¥y < 01, **p <.05, *p <.10. 
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Appendix Table A.26 


Effect of Program Assignment on Enrollment in and Completion of 
College-Level Math by Pell Recipient Status 


Treatment Status and Model Indicators (1) (2) (3) (4) 


Pell recipients only 


Program assignment 0.043 ** 0.043 *** 0.043 *** 0.043 *** 
(0.017) (0.017) (0.017) (0.017) 

Control mean 0.132 

Observations 1,763 1,763 1,763 1,763 


Non-Pell recipients only 


Program assignment 0.039* 0.038* 0.038* 0.036* 
(0.021) (0.021) (0.021) (0.019) 
Control mean 0.216 
Observations 1,633 1,633 1,633 1,633 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***D < 01, **p <.05, *p <.10. 
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Appendix Table A.27 


Effect of Program Assignment on Enrollment in and Completion of College-Level Math 


by Gender 
Treatment Status and Model Indicators (1) (2) (3) (4) 
Female students only 
Program assignment 0.064*** 0.062*** 0.063*** 0.059*** 
(0.018) (0.018) (0.018) (0.017) 
Control mean 0.146 
Observations 1,701 1,701 1,701 1,701 
Male students only 
Program assignment 0.012 0.013 0.014 0.014 
(0.018) (0.018) (0.018) (0.018) 
Control mean 0.195 
Observations 1,862 1,862 1,862 1,862 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***D < 01, **p <.05, *p <.10. 
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Appendix Table A.28 


Effect of Program Assignment on Placement in College-Level English by Race/Ethnicity 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Black students only 
Program assignment 0.382*** 0.383 *** 0.384*** 0.382*** 
(0.042) (0.042) (0.042) (0.039) 
Control mean 0.413 
Observations 477 477 477 477 


Hispanic students only 


Program assignment 0.328*** 0.324*** 0:323*#* 0:332"** 
(0.035) (0.035) (0.035) (0.033) 

Control mean 0.541 

Observations 574 574 574 574 


White students only 


Program assignment 0.220*** 0.221*** 0.218*** 0.203 *** 
(0.026) (0.026) (0.026) (0.024) 

Control mean 0.604 

Observations 1,030 1,030 1,030 1,030 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

**n < 01, **p <.05, *p <.10. 
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Appendix Table A.29 


Effect of Program Assignment on Placement in College-Level English 
by Pell Recipient Status 


Treatment Status and Model Indicators (1) (2) (3) (4) 


Pell recipients only 


Program assignment 0.307*** 0.309*** 0.308*** 0.296*** 
(0.024) (0.024) (0.024) (0.023) 
Control mean 0.485 
Observations 1,408 1,408 1,408 1,408 
Non-Pell recipients only 
Program assignment 0.278*** O:275*** 0.274*** 0.276*** 
(0.022) (0.022) (0.022) (0.021) 
Control mean 0.608 
Observations 1,340 1,340 1,340 1,340 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column | includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 


preparedness. 
***y < 01, **p <.05, *p <.10. 
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Appendix Table A.30 


Effect of Program Assignment on Placement in College-Level English by Gender 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Female students only 
Program assignment 0.309*** 0.312*** 0.312*** 0.306*** 
(0.024) (0.024) (0.024) (0.023) 
Control mean 0.535 
Observations 1,262 1,262 1,262 1,262 
Male students only 
Program assignment 0.289*** 0.289*** 0.290*** 0.284*** 
(0.022) (0.021) (0.021) (0.021) 
Control mean 0.55 
Observations 1,570 1,570 1,570 1,570 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

en < 01, **p <.05, *p <.10. 
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Appendix Table A.31 


Effect of Program Assignment on Enrollment in College-Level English by Race/Ethnicity 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Black students only 
Program assignment 0.326*** 0.325*** 0.324*** 0.320*** 
(0.043) (0.043) (0.043) (0.042) 
Control mean 0.409 
Observations 477 477 477 477 


Hispanic students only 


Program assignment 0.263 *** 0.265*** 0.264*** 0:273%** 
(0.038) (0.038) (0.038) (0.037) 

Control mean 0.515 

Observations 574 574 574 574 


White students only 


Program assignment O.17 148% 0.172*** 0.170*** 0.155*** 
(0.028) (0.028) (0.028) (0.027) 

Control mean 0.563 

Observations 1,030 1,030 1,030 1,030 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 
preparedness. 

**n < 01, **p <.05, *p <.10. 
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Appendix Table A.32 


Effect of Program Assignment on Enrollment in College-Level English 
by Pell Recipient Status 


Treatment Status and Model Indicators (1) (2) (3) (4) 


Pell recipients only 


Program assignment 0.272 *** 0.275 *** O:275*** 0.263 *** 
(0.025) (0.025) (0.025) (0.024) 
Control mean 0.446 
Observations 1,408 1,408 1,408 1,408 
Non-Pell recipients only 
Program assignment 0.202*** 0.195*** 0.195*** 0.196*** 
(0.025) (0.025) (0.025) (0.024) 
Control mean 0.580 
Observations 1,340 1,340 1,340 1,340 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column | includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 


preparedness. 
***y < 01, **p <.05, *p <.10. 
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Appendix Table A.33 


Effect of Program Assignment on Enrollment in College-Level English by Gender 


Treatment Status and Model Indicators (1) (2) (3) (4) 
Female students only 
Program assignment 0.233 *** 0.234*** 0.234*** 0.229*** 
(0.026) (0.026) (0.026) (0.025) 
Control mean 0.484 
Observations 1,262 1,262 1,262 1,262 
Male students only 
Program assignment 0:239*** 0.239*** 0.238*** 0.233%** 
(0.023) (0.023) (0.023) (0.022) 
Control mean 0.524 
Observations 1,570 1,570 1,570 1,570 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. The model from Column 1 includes only fixed effects for 
college and no additional controls. Column 2 includes fixed effects for colleges and controls for demographic 
indicators including race/ethnicity, gender, and age. Column 3 includes college fixed effects, controls for 
demographic indicators and proxies for income including Pell and TAP Grant recipient status. Column 4 includes all 
the previous controls plus calculated math and English algorithm values, which serve as proxies for college 


preparedness. 
***y < 01, **p <.05, *p <.10. 
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Appendix Table A.34 


Effect of Program Assignment on Enrollment in and Completion of College-Level English 


by Race/Ethnicity 
Treatment Status and Model Indicators (1) (2) (3) (4) 
Black students only 
Program assignment 0.190*** 0.183 *** 0.182*** 0.176*** 
(0.042) (0.043) (0.043) (0.042) 
Control mean 0.244 
Observations 477 477 477 477 
Hispanic students only 
Program assignment 0.149*** 0.149*** 0.145*** 0.154*** 
(0.041) (0.041) (0.041) (0.040) 
Control mean 0.344 
Observations 574 574 574 574 
White students only 
Program assignment 0.148*** 0.146*** 0.145*** 0.130*** 
(0.030) (0.030) (0.030) (0.030) 
Control mean 0.391 
Observations 1,030 1,030 1,030 1,030 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***n < 01, **p <.05, *p <.10. 
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Appendix Table A.35 


Effect of Program Assignment on Enrollment in and Completion of College-Level English 
by Pell Recipient Status 


Treatment Status and Model Indicators (1) (2) (3) (4) 


Pell recipients only 


Program assignment 0.177*** 0.177*** O.1778** 0.167*** 
(0.025) (0.025) (0.025) (0.024) 
Control mean 0.287 
Observations 1,408 1,408 1,408 1,408 
Non-Pell recipients only 
Program assignment 0.124*** O21*** 0.121*** 0.122*** 
(0.027) (0.027) (0.027) (0.026) 
Control mean 0.399 
Observations 1,340 1,340 1,340 1,340 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***n < 01, **p <.05, *p <.10. 
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Appendix Table A.36 


Effect of Program Assignment on Enrollment in and Completion of College-Level English 


by Gender 
Treatment Status and Model Indicators (1) (2) (3) (4) 
Female students only 
Program assignment 0.168*** 0.169*** 0.170*** 0.165*** 
(0.027) (0.027) (0.027) (0.026) 
Control mean 0.344 
Observations 1,262 1,262 1,262 1,262 
Male students only 
Program assignment 0.139*** 0.139*** 0.139*** 0.134*** 
(0.024) (0.024) (0.024) (0.024) 
Control mean 0.333 
Observations 1,570 1,570 1,570 1,570 
College fixed effects YES YES YES YES 
Demographic indicators NO YES YES YES 
Income indicators NO NO YES YES 
College preparedness measures NO NO NO YES 


NOTES: Robust standard errors shown in parentheses. Completion is defined as earning a grade of C or better. The 
model from Column 1 includes only fixed effects for college and no additional controls. Column 2 includes fixed 
effects for colleges and controls for demographic indicators including race/ethnicity, gender, and age. Column 3 
includes college fixed effects, controls for demographic indicators and proxies for income including Pell and TAP 
Grant recipient status. Column 4 includes all the previous controls plus calculated math and English algorithm values, 
which serve as proxies for college preparedness. 

***y < 01, **p <.05, *p <.10. 
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Appendix Figure A.1 


Relationship Between Minimum Detectable Effect (MDE) and Sample Size: 
Binary Outcomes 
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NOTES: This figure plots the MDE for binary outcomes based on 80 percent power, a 5 percent 
significance level, and a two-tailed test, and assumes (conservatively) that the sample proportion randomly 
assigned for treatment is equal to 0.5. The figure presents two scenarios: (1) the top (dark) line calculates 
the MDE when a = 0.5, and (2) the bottom (light) line calculates the MDE when z = 0.25, where z is equal 
to the proportion of successful outcomes. (For binary outcomes, the MDE is symmetric at z = 0.5.) 
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